Collation device, collation method, collation program, machine learning method, and machine learning device

Abstract

To provide a technique for improving the accuracy of collation between elements.SOLUTION: A collation device according to an aspect of the present invention acquires a probability distribution of a distance between a first sample of a first input including a plurality of elements and a second sample of a second input including a plurality of elements, gives the acquired probability distribution to a collation model, executes arithmetic processing of the collation model to acquire a collation result between the elements of the first sample and the second sample from the collation model, and outputs information on the acquired collation result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a matching device, a matching method, a matching program, a machine learning method, and a machine learning device. [Background technology]

[0002] In recent years, there has been progress in the development of technologies for matching elements between multiple inputs. For example, Non-Patent Document 1 proposes a matching method that calculates the distance between each element included in each sample of point cloud data and matches the elements of each sample according to the calculated distance (i.e., infers the combination of elements). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-116884 [Non-patent literature]

[0004] [Non-Patent Document 1] Yang Li, et al. “Lepard: Learning partial point cloud matching in rigid and deformable scenes”, [online], [Retrieved June 10, 2020], Internet<URL:https: / / arxiv.org / abs / 2111.12591> [Non-patent document 2] Shusaku Sone, et al. “WeaveNet for Approximating Two-Sided Matching Problems”, [online], [Retrieved June 10, 2024], Internet<URL:https: / / arxiv.org / abs / 2310.12515> [Non-patent document 3] Andy Zeng, et al. "3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions", [online], [Reiwa June 10, 2016], インターネット<URL:https: / / arxiv.org / abs / 1603.08182>

Non-licensed Document 4

Non-licensed Document 5

Non-licensed Document 6

Non-licensed Document 7

[0005] The present inventors have found that the above-described conventional methods have the following problems. That is, in the conventional techniques, a deterministic algorithm is used as the matching algorithm. The deterministic algorithm is configured to infer a combination of elements without considering the uncertainty of the distance between each element (i.e., without considering the possibility that the given distance may deviate from the true value). When the accuracy of the distance value between each element is high, using a deterministic algorithm for matching between elements can be expected to achieve high accuracy. However, distance values ​​cannot always be obtained with high accuracy. For example, when the true distance value cannot be obtained, the accuracy of the distance value between each element may be low. When a deterministic algorithm is used for matching between elements in a situation where the accuracy of the distance value between each element is low, it may be difficult to ensure the accuracy of the matching due to the influence of the low accuracy of the distance.

[0006] In one aspect, the present invention has been made in view of the above circumstances, and an object of the present invention is to provide a technique for improving the accuracy of matching between elements. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, the present invention employs the following configurations. Note that the following configurations of the invention can be combined as appropriate.

[0008] A matching device according to one aspect of the present invention includes a control unit configured to acquire a probability distribution of distances between a first sample of a first input including a plurality of elements and a second sample of a second input including a plurality of elements, provide the acquired probability distribution to a matching model, and execute a calculation process of the matching model to acquire a matching result between the elements of the first sample and the second sample, and output information related to the acquired matching result.

[0009] In this configuration, the matching model is configured to derive matching results between elements from a probability distribution of the distances between each sample. The probability distribution of the distances may indicate uncertainty about the distances. A solver for a stochastic optimization problem can be used for the matching model. With this configuration, by using such a matching model, it is possible to infer combinations of elements between samples while taking into account the possibility (uncertainty) that the given distances are incorrect. As a result, it is expected that matching accuracy will be ensured even in situations where the accuracy of the given distances is low. Therefore, with this configuration, it is expected that the accuracy of matching between elements will be improved.

[0010] In the matching device according to the above aspect, the matching model may be configured by differentiable operations. The acquiring of the probability distribution of distances may be configured by calculating, using a feature extractor trained by machine learning, feature amounts of the elements included in the first sample and the second sample, and calculating the probability distribution of the distances between the first sample and the second sample from the calculated feature amounts of the elements included in the first sample and the second sample.

[0011] Training by machine learning is performed by adjusting (optimizing) the values ​​of the calculation parameters included in the model using training samples. For example, an algorithm such as backpropagation may be used as the training algorithm. In order to solve the optimization problem in this training, the calculations included in the model must be differentiable. In this configuration, since the matching model is composed of differentiable calculations, when machine learning the feature extractor, it is possible to solve the optimization problem of the feature extractor by tracing back from the output of the matching model. In other words, machine learning of the feature extractor can be performed in conjunction with the calculations of the matching model. Compared to when machine learning of the feature extractor is performed separately, by performing machine learning of the feature extractor in conjunction with the calculations of the matching model, By taking into account the computation of the matching model, it is possible to obtain a better optimal solution for the feature extractor for the matching task. With this configuration, by using a feature extractor trained by such machine learning, it is expected that the accuracy of matching will be improved.

[0012] In the matching device according to the above aspect, the matching model may be a machine learning model trained by machine learning integrated with the feature extractor. When machine learning is performed separately for the feature extractor and the matching model, the feature extractor and the matching model may each fall into a local optimum solution. In contrast, with this configuration, by performing machine learning for the matching model and the feature extractor integratedly, it is possible to obtain a better optimal solution for the matching task for each of the feature extractor and the matching model. Therefore, by using a feature extractor and a matching model trained by such machine learning, improved matching accuracy can be expected.

[0013] In the matching device according to the above aspect, calculating the probability distribution of the distances from the feature quantities of the elements may include calculating an evaluation value between the elements of the first sample and the second sample from the calculated feature quantities, selecting one or more combinations from the combinations of the elements of the first sample and the second sample based on the calculated evaluation value, and calculating a probability distribution of the distances between the elements of each of the selected one or more combinations from the calculated feature quantities. The probability distribution of the distances between the first sample and the second sample may be calculated from the probability distribution of the distances between the elements of each of the selected one or more combinations. With this configuration, by selecting targets for matching, it is possible to reduce the processing load (computational cost).

[0014] In the matching device according to the above aspect, the first input may be first point cloud data, and the second input may be second point cloud data. The matching between elements may be a matching between points included in the first point cloud data and points included in the second point cloud data. With this configuration, improved matching accuracy can be expected when matching points between point cloud data.

[0015] In the matching device according to the above aspect, the first input may be a first image, and the second input may be a second image. The matching between elements may be a matching between points included in the first image and points included in the second image. With this configuration, improved matching accuracy can be expected when matching points (feature points, pixels, etc.) between images.

[0016] In the matching device according to the above aspect, the first input may be a first set of people, and the second input may be a second set of people. The matching between elements may be a matching between people included in the first set of people and people included in the second set of people. With this configuration, improved matching accuracy can be expected when matching people between sets of people.

[0017] In the matching device according to the above aspect, the first input may be a group of people, and the second input may be a group of objects. The matching between elements may be a matching between people included in the group of people and objects included in the group of objects. With this configuration, improved matching accuracy can be expected when matching people and objects.

[0018] In the matching device according to the above aspect, the first input may be a first object set, and the second input may be a second object set. The matching between elements may be a matching between an object included in the first object set and an object included in the second object set. With this configuration, improvement in matching accuracy can be expected when matching objects between object sets.

[0019] The present invention is not limited to the above-described matching (inference) step. One aspect of the present invention is a feature extractor and a matching device that can be used in the matching device according to at least one of the above-described aspects. The method may be a machine learning method for generating at least one of the joint models.

[0020] For example, a machine learning method according to one aspect of the present invention may include a computer-implemented step of controlling machine learning of a feature extractor linked to a matching model and a step of outputting a result of the machine learning. The feature extractor may be configured to calculate a feature value for each element included in an input. The matching model may be configured to derive a matching result between elements included in a first input and a second input from a probability distribution of distances between the first input and the second input using a differentiable operation. The machine learning may include using the feature extractor to calculate a feature value for each element included in a first training sample of the first input and a second training sample of the second input, calculating a probability distribution of distances between the first training sample and the second training sample from the calculated feature values ​​of each element included in the first training sample and the second training sample, providing the calculated probability distribution of distances to the matching model and performing a computation process of the matching model to derive a matching result between the elements of the first training sample and the second training sample, and training the feature extractor so that the derived matching result approaches true values ​​corresponding to the first training sample and the second training sample. With this configuration, compared to when the machine learning of the feature extractor is performed separately, by performing the machine learning of the feature extractor in conjunction with the calculation of the matching model, it is possible to obtain a better optimal solution of the feature extractor for the matching task by taking into account the calculation of the matching model. Therefore, by using the generated trained feature extractor for the matching task, it is possible to expect an improvement in the accuracy of matching.

[0021] In the machine learning method according to the above aspect, training the feature extractor may be configured by jointly training the feature extractor and the matching model. With this configuration, jointly executing machine learning for the matching model and the feature extractor may potentially result in better optimal solutions for the feature extractor and the matching model for the matching task. Therefore, by using the feature extractor and the matching model trained by such machine learning for the matching task, improved matching accuracy can be expected.

[0022] In the machine learning method according to the above aspect, calculating the probability distribution of the distances from the feature values ​​of the elements may include calculating an evaluation value between the elements of the first training sample and the second training sample from the calculated feature values, selecting one or more combinations from the combinations of the elements of the first training sample and the second training sample based on the calculated evaluation value, and calculating a probability distribution of the distances between the elements of each of the selected one or more combinations from the calculated feature values. The probability distribution of the distances between the first training sample and the second training sample may be formed from the probability distribution of the distances between the elements of each of the selected one or more combinations. With this configuration, by selecting targets for matching, it is possible to reduce the processing load (computational cost).

[0023] In the machine learning method according to the above aspect, the first input may be first point cloud data, and the second input may be second point cloud data. The matching between elements may be matching between points included in the first point cloud data and points included in the second point cloud data. With this configuration, improvement in matching accuracy can be expected when matching points between point cloud data.

[0024] In the machine learning method according to the above aspect, the first input may be a first image, and the second input may be a second image. The matching between elements may be matching between points included in the first image and points included in the second image. This configuration is expected to improve matching accuracy when matching points between images.

[0025] In the machine learning method according to the above aspect, the first input may be a first set of people, and the second input may be a second set of people. The matching between elements may be matching between people included in the first set of people and people included in the second set of people. With this configuration, improvement in matching accuracy can be expected when matching people between sets of people.

[0026] In the machine learning method according to the above aspect, the first input may be a set of people, and the second input may be a set of objects. The matching between elements may be a matching between people included in the set of people and objects included in the set of objects. With this configuration, improved matching accuracy can be expected when matching people and objects.

[0027] In the machine learning method according to the above aspect, the first input may be a first set of entities, and the second input may be a second set of entities. The matching between elements may be a matching between entities included in the first set of entities and entities included in the second set of entities. This configuration can be expected to improve matching accuracy when matching entities between entity sets.

[0028] Note that the present invention is not limited to the above-described verification device (information processing device). As another aspect of the verification device according to each of the above aspects, one aspect of the present invention may be an information processing method (verification method) that realizes all or part of each of the above configurations, or may be a program, or may be a storage medium readable by a machine such as a computer that stores such a program. A storage medium readable by a machine such as a computer may be a non-transitory medium that stores information such as a program by electrical, magnetic, optical, mechanical, or chemical action. The non-transitory storage medium may include a storage medium (CD, DVD, semiconductor memory, etc.), an auxiliary storage device of a computer, an external storage device connected to a computer, etc.

[0029] Furthermore, the machine learning method according to each of the above aspects is a method for generating a trained feature extractor (machine learning model). The machine learning method according to one of the above aspects is a method for further generating a trained matching model. The form of the present invention is not limited to the above machine learning method (information processing method). As another mode of the machine learning method according to each of the above aspects, one aspect of the present invention may be an information processing device (machine learning device) that realizes all or part of each of the above configurations, or may be a program, or may be a storage medium readable by a machine such as a computer that stores such a program.

[0030] For example, a matching method according to one aspect of the present invention may be an information processing method in which a computer executes the steps of: acquiring a probability distribution of distances between a first sample of a first input including a plurality of elements and a second sample of a second input including a plurality of elements; providing the acquired probability distribution to a matching model and performing computational processing of the matching model to acquire a matching result between the elements of the first sample and the second sample; and outputting information relating to the acquired matching result.

[0031] Furthermore, for example, a matching program according to one aspect of the present invention may be a program for causing a computer to execute the steps of: acquiring a probability distribution of the distance between a first sample of a first input including a plurality of elements and a second sample of a second input including a plurality of elements; providing the acquired probability distribution to a matching model and executing computational processing of the matching model to acquire a matching result between the elements of the first sample and the second sample; and outputting information related to the acquired matching result.

[0032] Furthermore, for example, a machine learning device according to an aspect of the present invention may include a control unit. The control unit may be configured to control machine learning of a feature extractor linked to a matching model and to output the results of the machine learning. The feature extractor may be configured to calculate a feature amount of each element included in the input. The matching model may be configured to perform a differentiable operation. and deriving a matching result between elements included in the first input and the second input from a probability distribution of distances between the first input and the second input. The machine learning may include: using the feature extractor to calculate a feature amount of each element included in a first training sample of the first input and a second training sample of the second input; calculating a probability distribution of distances between the first training sample and the second training sample from the calculated feature amount of each element included in the first training sample and the second training sample; providing the calculated probability distribution of distances to the matching model and performing a calculation process of the matching model to derive a matching result between the elements of the first training sample and the second training sample; and training the feature extractor so that the derived matching result approaches true values ​​corresponding to the first training sample and the second training sample. [Effects of the Invention]

[0033] According to the present invention, it is possible to expect improvement in matching accuracy. [Brief explanation of the drawings]

[0034] [Figure 1] FIG. 1 shows a schematic diagram of an example of a situation in which the present invention is applied. [Figure 2] FIG. 2 shows a schematic diagram of an example of a method for obtaining a probability distribution. [Figure 3] FIG. 3 shows a schematic diagram of an example of a method for calculating a probability distribution. [Figure 4] FIG. 4 schematically shows another example of a method for calculating a probability distribution. [Figure 5] FIG. 5 shows a schematic diagram of an example of a system configuration. [Figure 6] FIG. 6 shows a schematic diagram of a first example to which the present invention is applied. [Figure 7] FIG. 7 is a diagram showing an example of a second case to which the present invention is applied. [Figure 8] FIG. 8 shows a schematic diagram of a third example to which the present invention is applied. [Figure 9] FIG. 9 is a schematic diagram showing an example of a fourth case to which the present invention is applied. [Figure 10] FIG. 10 is a schematic diagram showing an example of a fifth case to which the present invention is applied. [Figure 11] FIG. 11 is a diagram illustrating an example of a hardware configuration of the verification device. [Figure 12] FIG. 12 schematically illustrates an example of the hardware configuration of a machine learning device. [Figure 13] FIG. 13 is a diagram illustrating an example of the software configuration of the verification device. [Figure 14] FIG. 14 schematically illustrates an example of the software configuration of the machine learning device. [Figure 15] FIG. 15 is a flowchart illustrating an example of a processing procedure of the machine learning device. [Figure 16] FIG. 16 is a flowchart illustrating an example of a processing procedure of the verification device. [Figure 17] FIG. 17 shows the evaluation results of the comparative example and the working example in the first experimental example. [Figure 18] FIG. 18 shows the measurement results of the comparative example and the example in the second experimental example. [Figure 19]FIG. 19 shows the evaluation results of the comparative example and the working example in the third experimental example. [Figure 20] FIG. 20 shows the evaluation results of the comparative example and the working example in the third experimental example. [Figure 21] FIG. 21 shows the evaluation results of the comparative example and the working example in the fourth experimental example. [Figure 22] FIG. 22 shows the evaluation results of the comparative example and the working example in the fifth experimental example. [Figure 23] FIG. 23 shows the evaluation results of the comparative example and the working example in the fifth experimental example. [Figure 24] FIG. 24 shows an example of a failed match in the comparative example in the fifth experimental example. [Figure 25] FIG. 25 shows an example of successful matching in the comparative example in the fifth experimental example. [Figure 26] FIG. 26 shows an example of a failed match in the working example of the fifth experimental example. [Figure 27] FIG. 27 shows an example of successful matching in the embodiment in the fifth experimental example. DETAILED DESCRIPTION OF THE INVENTION

[0035] An embodiment according to one aspect of the present invention will be described below with reference to the drawings. However, the embodiment described below is merely an example of the present invention in all respects. Various improvements or modifications may be made without departing from the scope of the present invention. In carrying out the present invention, a specific configuration according to the embodiment may be appropriately adopted. Note that although the data appearing in this embodiment is explained in natural language, more specifically, it may be explained in pseudo-language that can be recognized by a computer. It is specified by words, commands, parameters, machine language, etc.

[0036] §1 Application Examples FIG. 1 schematically illustrates an example of a scenario in which the present invention is applied. A matching device 1 according to this embodiment is one or more computers configured to perform a task of matching elements between two inputs. The matching task is to infer combinations of elements (combinatorial optimization). In this embodiment, the matching device 1 acquires a probability distribution 40 of the distance between a first sample 30 of a first input including multiple elements and a second sample 35 of a second input including multiple elements. The matching device 1 provides the acquired probability distribution 40 to a matching model 50, and executes the calculation process of the matching model 50. As a result, the matching device 1 acquires a matching result between the elements of the first sample 30 and the second sample 35 from the matching model 50. The matching device 1 outputs information related to the acquired matching result.

[0037] In this embodiment, the matching model 50 is configured to derive a matching result between elements from a probability distribution 40 of the distance between each sample (30, 35). The probability distribution 40 of the distance may indicate uncertainty of the distance. According to this embodiment, by using such a matching model 50, it is possible to infer the combination of elements between the samples (30, 35) while taking into account the possibility (uncertainty) that the given distance is incorrect. As a result, it is expected that matching accuracy will be ensured even in situations where the accuracy of the given distance is low. Therefore, according to this embodiment, it is expected that the accuracy of matching between elements will be improved.

[0038] [Probability Distribution] The probability distribution of the distance is configured to be able to express the uncertainty of the distance. In one example, the uncertainty can be expressed by multiple numerical values. Therefore, the probability distribution may be configured so that multiple numerical values ​​are assigned to one distance between elements. In other words, the probability distribution may be configured to express one distance between elements by multiple numerical values. As a typical example, a normal distribution may be adopted as the probability distribution, and one distance between elements may be expressed by two numerical values, a mean and a variance. The probability distribution of the distance may be configured by a distribution known in probability theory or statistics, in addition to the normal distribution. In another example, the probability distribution of the distance may be configured by any configuration other than a known distribution.

[0039] The distance between samples may be appropriately configured to include the distance between at least some of the elements of each sample, i.e., the distance between at least some combinations of elements between each sample. In one example, the distance between samples may be configured as the distance between all of the elements of each sample (i.e., the distance between all of the combinations of elements between the samples). In another example, the distance between samples may be configured as the distance between some of the elements of each sample (i.e., the distance between some of the combinations of elements between the samples). The distance between elements may be arbitrarily defined. The definition of the distance between elements may employ well-known definitions such as Euclidean distance and cosine distance, or a definition specific to an embodiment. The method of acquiring the probability distribution of the distances is not particularly limited and may be appropriately determined depending on the embodiment. At least a portion of the calculations for acquiring the probability distribution of the distances 40 may be executed in the matching device 1 or may be executed in a computer other than the matching device 1.

[0040] 2 is a diagram illustrating an example of a method for obtaining a probability distribution 40 according to the present embodiment. As illustrated in FIG. 2, in one example, a feature extractor 60 may be used to obtain the distance probability distribution 40. Specifically, obtaining the distance probability distribution 40 involves using the feature extractor 60 to calculate feature quantities (31, 36) of each element included in the first sample 30 and the second sample 35, respectively, and extracting the distance probability distribution 40 from the calculated feature quantities (31, 36) of each element included in the first sample 30 and the second sample 35. The feature 31 of each element included in the first sample 30 may be referred to as a "first feature," and the feature 36 of each element included in the second sample 35 may be referred to as a "second feature." The number of dimensions of the feature (31, 36) of each element may be determined appropriately depending on the embodiment. The probability distribution 40 of the distance between the elements of each sample (30, 35) may be calculated appropriately from each feature (31, 36). At least a part of the calculation to calculate the probability distribution 40 of the distance from each feature (31, 36) may be executed in the matching device 1, or may be executed in a computer other than the matching device 1.

[0041] FIG. 3 schematically illustrates an example of a method for calculating a probability distribution 40 according to an embodiment. In the example of FIG. 3, a scenario is assumed in which the first sample 30 contains three elements and the second sample 35 contains four elements (the total number of element combinations is 12). Each feature 310 is an example of a feature 31 calculated for each element in the first sample 30 using the feature extractor 60. Each feature 360 ​​is an example of a feature 36 calculated for each element in the second sample 35 using the feature extractor 60. Note that the number of elements in each of the first sample 30 (first input) and the second sample 35 (second input) is not limited to this example and may be determined appropriately depending on the embodiment. The number of elements in each of the first sample 30 and the second sample 35 may be the same or different, as in the example of FIG. 3.

[0042] In the example of FIG. 3 , distribution 315 may be generated by duplicating the set of feature quantities 310 of the first sample 30 the number of times corresponding to the number of elements included in the second sample 35 (four in the figure). Furthermore, distribution 365 may be generated by duplicating the set of feature quantities 360 of the second sample 35 the number of times corresponding to the number of elements included in the first sample 30 (three in the figure). Each distribution (315, 365) may correspond to a third-order tensor whose first axis corresponds to the elements included in the first sample 30, whose second axis corresponds to the elements included in the second sample 35, and whose third axis corresponds to each feature quantity (310, 360). A probability distribution 400 of the distance between each element may be calculated by combining (e.g., using a cat operation) each distribution (315, 365) on the axis of each feature quantity (310, 360). 3, the probability distribution 400 of the distance between the n-th element of the first sample 30 and the m-th element of the second sample 35 may be configured as a combination (e.g., a simple combination) of the feature 310 of the n-th element and the feature 360 ​​of the m-th element. This probability distribution 400 of the distance is an example of the probability distribution 40. The matching device 1 may provide the set of probability distributions 400 to the matching model 50 and execute the calculation process of the matching model 50 to obtain a matching result between the elements of each sample (30, 35).

[0043] (Selected) 3, a distance probability distribution 40 (probability distribution 400) is obtained for all combinations of elements of the first sample 30 and the second sample 35, and the obtained probability distribution 40 (probability distribution 400) is provided to the matching model 50. In other words, all combinations of elements of the first sample 30 and the second sample 35 are subject to matching calculation by the matching model 50. However, if all combinations are subject to matching calculation, the processing load (computation cost) on the matching model 50 increases as the number of elements included in each sample (30, 35) increases.

[0044] Therefore, in one example of this embodiment, one or more combinations that are estimated to have a high probability of matching may be selected from the combinations of elements of each sample (30, 35), thereby narrowing down the combinations to be used in the matching calculation. The selection method is not particularly limited and may be selected appropriately depending on the embodiment. For example, calculating the probability distribution 40 of the distance from the feature quantities (31, 36) of each element may involve calculating an evaluation value between each element of the first sample 30 and the second sample 35 from the calculated feature quantities (31, 36), and then calculating the probability distribution 40 based on the calculated evaluation value. Based on this, the probability distribution 40 of the distance between the elements of the first sample 30 and the second sample 35 may be constructed by selecting one or more combinations from the combinations of elements of the first sample 30 and the second sample 35, and calculating a probability distribution 40 of the distance between the elements of each of the selected one or more combinations from the calculated feature amounts (31, 36). Accordingly, the probability distribution 40 of the distance between the first sample 30 and the second sample 35 may be constructed from the probability distribution 40 of the distance between the elements of each of the selected one or more combinations.

[0045] The evaluation value may be calculated together with the distance (i.e., the distance may be used as the evaluation value), or may be calculated separately from the distance. The method of calculating the evaluation value may be defined appropriately depending on the embodiment. The evaluation value may be calculated from at least a portion of the feature quantities (31, 36) of each corresponding element of each sample (30, 35). The distance probability distribution 40 may also be calculated from at least a portion of the feature quantities (31, 36) of each corresponding element of each sample (30, 35). The portion of the feature quantities (31, 36) used to calculate the evaluation value and the portion of the feature quantities (31, 36) used to calculate the distance probability distribution 40 may or may not overlap. When the latter method is adopted, for example, the evaluation value may be calculated from a portion of the feature quantities (31, 36) of each corresponding element, and the distance probability distribution 40 may be calculated from at least a portion of the remaining feature quantities (31, 36). The calculated evaluation value may or may not be used to calculate the distance probability distribution 40. When calculating distance as an evaluation value, the distance value used for selection may be included in the calculation of probability distribution 40, or may be excluded from the calculation of probability distribution 40 (i.e., it may be used only as an evaluation value).

[0046] FIG. 4 schematically illustrates another example of a method for calculating a probability distribution 40 according to an embodiment. In the example of FIG. 4, similar to FIG. 3, a scenario is assumed in which the first sample 30 contains three elements and the second sample 35 contains four elements (the total number of element combinations is 12). Each feature 311 is an example of the feature 31 calculated for each element in the first sample 30 using the feature extractor 60. The feature 311 for each element in the first sample 30 includes a first portion 3111 and a second portion 3112. Each portion (3111, 3112) may be composed of one or more numerical values. Each feature 361 is an example of the feature 36 calculated for each element in the second sample 35 using the feature extractor 60. The feature 361 for each element in the second sample 35 includes a first portion 3611 and a second portion 3612. Each portion (3611, 3612) may also be composed of one or more numerical values. The second part (3112, 3612) is an example of a portion of the features (31, 36) used to calculate the evaluation value, and the first part (3111, 3611) is an example of the remaining portion of the features (31, 36) used to calculate the probability distribution 40.

[0047] 4, the evaluation value 391 between each element may be calculated from the second portion 3112 of the feature 311 of the first sample 30 and the second portion 3612 of the feature 361 of the second sample 35. The evaluation value 391 between the n-th element of the first sample 30 and the m-th element of the second sample 35 may be calculated from the second portion 3112 of the feature 311 of the n-th element of the first sample 30 and the second portion 3612 of the feature 361 of the m-th element of the second sample 35. In one example, each evaluation value 391 may be calculated from the second portion (3112, 3612) of the feature (311, 361) of each element of each sample (30, 35) using a known distance calculation method such as Euclidean distance or cosine distance.

[0048] Next, one or more combinations may be selected from the combinations of elements of each sample (30, 35) based on the evaluation value 391 of each combination. The method of selection according to the evaluation value 391 may be defined appropriately depending on the embodiment. In a typical example, one or more combinations may be selected according to the result of comparison between the evaluation value 391 and a threshold (less than the threshold, equal to or less than the threshold, equal to or greater than the threshold, exceeding the threshold, etc.). The threshold may be set appropriately. The number of combinations to be selected may be fixed or variable. In the example of FIG. 4, A scenario is assumed in which five combinations of elements are selected from the combinations of elements in the first sample 30 and the second sample 35: first element-second element, first element-fourth element, second element-first element, third element-third element, and third element-fourth element.

[0049] For the first sample 30, the first portion 3111 of the feature 311 may be selected, duplicated, or deleted corresponding to the selected combination with the element of the second sample 35. The first portion 3111 of the elements of the first sample 30, which are paired as the selected combination with one element of the second sample 35, may be selected to correspond to the paired element of the second sample 35. The first portion 3111 of the elements of the first sample 30, which are paired as two or more elements of the second sample 35, may be duplicated the number of times equal to the number of paired elements of the second sample 35. The first portion 3111 of the elements that are not paired with the element of the second sample 35 (i.e., do not belong to the selected combination) may be deleted. In a simple example, the set of the first portion 3111 of the feature 311 may be duplicated the number of times equal to the number of elements included in the second sample 35, and components corresponding to the combinations that were not selected may be deleted from the distribution obtained by duplication. This may generate a distribution 316 of the feature 311. Similarly, for the second sample 35, a first portion 3611 of the feature 361 may be selected, duplicated, or deleted corresponding to a selected combination with elements of the first sample 30. This may generate a distribution 366 of the feature 361.

[0050] Then, by combining (for example, cat operation) the distribution 316 of the first sample 30, the distribution 396 of the evaluation value 391 (distance) of the selected combination, and the distribution 366 of the second sample 35 on the axis of each feature (311, 361), the probability of each of the one or more selected combinations is calculated. A probability distribution 401 may be calculated. That is, in the example of FIG. 4 , when a combination of the n-th element of the first sample 30 and the m-th element of the second sample 35 is selected, the probability distribution 401 of the distance between the n-th element of the first sample 30 and the m-th element of the second sample 35 may be configured as a combination (e.g., a simple combination) of the first portion 3111 of the feature 311 of the n-th element, the evaluation value 391 (distance) between the n-th element and the m-th element, and the first portion 3611 of the feature 361 of the m-th element. The probability distribution 401 calculated for the selected combination is an example of the probability distribution 40 of the distance between the elements of the selected combination. The matching device 1 may provide the set of the obtained probability distributions 401 to the matching model 50 and execute the calculation process of the matching model 50 to obtain a matching result between the elements of each sample (30, 35). According to this example of the present embodiment, it is possible to narrow down the combinations of elements to be calculated by the matching model 50, thereby reducing the processing load (calculation cost) on the matching model 50.

[0051] (summary) 4, the distance probability distribution 401 is calculated from the first portion (3111, 3611) of each feature (311, 361). That is, by summarizing the feature (31, 36) of each element, the feature (31, 36) used to calculate the probability distribution 40 is narrowed down to a portion. This makes it possible to reduce the amount of data in the probability distribution 40. Therefore, by summarizing the feature (31, 36) of each element, it is possible to reduce the processing load (computational cost) in the matching model 50.

[0052] (others) The method for calculating the distance probability distribution 40 does not have to be limited to the method using the feature extractor 60. In another example, multiple observations may be made for the distance between each element of the first sample 30 and the second sample 35, and the obtained observation results (a set of distances) may be used as the probability distribution 40. For example, the probability distribution 40 may be calculated from the obtained observation results using a method such as kernel density estimation or a non-parametric method. The observation may be actual or may be a simulation. One or more observations may be made, and the obtained observation results may be calculated under arbitrary conditions (for example, random, changing some parameters according to a rule, etc.). By expanding the data, multiple observation results may be obtained. The formula for calculating the distance probability distribution 40 may be given manually.

[0053] [Matching Model] The matching model 50 is configured to derive a matching result between elements included in the first input and the second input from a probability distribution of the distance between the first input and the second input. A solver for a stochastic optimization problem can be used for the matching model 50. In one example, the matching model 50 may be configured with differentiable operations. In another example, the matching model 50 may include non-differentiable operations.

[0054] In one example, the matching model 50 may be configured by a machine learning model and may be generated based on machine learning. The machine learning model has one or more calculation parameters that can be adjusted by machine learning. The one or more calculation parameters are used to calculate the desired inference. The machine learning model may be configured by, for example, a neural network, a support vector machine, a regression model, or other functional expressions (calculation models). When the matching model 50 includes a neural network, the structure of the neural network is not particularly limited and may be determined appropriately depending on the embodiment. The structure of the neural network may be specified by, for example, the number of layers from the input layer to the output layer, the type of each layer, the number of nodes (neurons) included in each layer, the connection relationships between the nodes in each layer, etc. The neural network may include any mechanism such as a recurrent structure, a self-attention mechanism, or an autoregressive model. The neural network may include any layer such as a fully connected layer, a convolutional layer, a pooling layer, a deconvolutional layer, an unpooling layer, a normalization layer, a dropout layer, or a long short-term memory (LSTM). Furthermore, the neural network may include any type of model, such as a diffusion model, a transformer model, or a generative model. The connection weights between nodes included in the neural network and the threshold value of each node are examples of calculation parameters. As long as the matching model 50 can accept input of the distance probability distribution 40, the configuration of the matching model 50 is not particularly limited and may be determined appropriately depending on the embodiment. The matching model 50 may be configured using a known machine learning model, such as WeaveNet, proposed in Patent Document 1 and Non-Patent Document 2. The machine learning method may be selected appropriately depending on the embodiment of the employed machine learning model (e.g., backpropagation).

[0055] [Feature Extractor] The feature extractor 60 is configured to calculate a feature amount for each element included in the input. As long as the feature extractor 60 can calculate the feature amount for each element, the configuration of the feature extractor 60 is not particularly limited and may be determined appropriately depending on the embodiment. The feature amount may be composed of one or more numerical values ​​so as to indicate the feature of the element.

[0056] In one example, the feature extractor 60 may be configured with a machine learning model and may be generated based on machine learning. The machine learning model may be configured with, for example, a neural network, a support vector machine, a regression model, or other functional formulas (computational models). As with the matching model 50, when the feature extractor 60 includes a neural network, the structure of the neural network is not particularly limited and may be determined appropriately depending on the embodiment. The machine learning method in the feature extractor 60 may also be selected appropriately depending on the embodiment, such as the machine learning model to be adopted.

[0057] FIG. 5 shows a schematic diagram of an example of a system configuration according to this embodiment. The system according to this embodiment includes a matching device 1 and a machine learning device 2. The machine learning device 2 is one or more computers configured to generate a feature extractor 60 trained by machine learning. The machine learning device 2 controls the machine learning of the feature extractor 60. The machine learning device 2 then outputs the results of the machine learning. Upon completion of the machine learning process, the trained feature extractor 60 The feature extractor 60 trained by machine learning may be provided to the matching device 1 by the machine learning device 2 in any manner and at any timing. The matching device 1 may use the feature extractor 60 trained by machine learning to calculate the feature amounts (31, 36) of each element included in each sample (30, 35).

[0058] In one example of this embodiment, when the matching model 50 is configured with differentiable operations, the machine learning of the feature extractor 60 may be linked with the matching model 50. Accordingly, the machine learning may include: using the feature extractor 60 to calculate feature quantities (34, 39) of each element included in the first training sample 33 of the first input and the second training sample 38 of the second input; calculating a probability distribution 45 of the distance between the first training sample 33 and the second training sample 38 from the feature quantities (34, 39) of each element included in the first training sample 33 and the second training sample 38; providing the calculated probability distribution 45 of the distance to the matching model 50 and executing the operation process of the matching model 50 to derive a matching result between the elements of the first training sample 33 and the second training sample 38; and training the feature extractor 60 so that the derived matching result approaches a true value 70 corresponding to the first training sample 33 and the second training sample 38.

[0059] A combination of the first training sample 33, the second training sample 38, and the true value 70 may be referred to as a dataset. The true value 70 may be configured as appropriate to indicate a correct answer for matching the corresponding first training sample 33 and second training sample 38. A plurality of datasets may be used to train the feature extractor 60. Each dataset may be collected as appropriate depending on the embodiment. Each dataset may be collected in the machine learning device 2, or may be collected in a computer other than the machine learning device 2. Machine learning is the use of training samples to adjust (optimize) the values ​​of calculation parameters. A training method in machine learning may be determined as appropriate depending on the embodiment.

[0060] For example, if the feature extractor 60 is configured as a neural network, the training method may employ an error backpropagation method. Accordingly, in the training process in machine learning, a matching result for each training sample (33, 38) may be derived by performing a forward calculation process. The error between the derived matching result and the corresponding true value 70 may be calculated. Then, the calculated error may be backpropagated to adjust the values ​​of the calculation parameters of the feature extractor 60. For example, this adjustment process may be repeated until a predetermined condition is met, such as the error being less than a threshold or a predetermined number of repetitions, to generate a trained feature extractor 60. If the matching model 50 is configured as a machine learning model and is trained together with the feature extractor 60, the values ​​of the calculation parameters of the matching model 50 may also be adjusted during this backpropagation process. This allows the trained feature extractor 60 and the trained matching model 50 to be generated.

[0061] In the conventional method, the computational process of the matching task is expressed by the following Equations 1 and 2.

[0062]

number

number

[0063] P and Q represent the inputs. D represents the distance between the elements of each sample. e indicates the distance calculation result. ε indicates the feature extraction operation, c indicates the distance operation, M indicates the matching solver, m indicates the matching operation, and M e indicates the matching result. F in Equation 1 is a stochastic process modeled by feature extraction (ε) and distance calculation (c). F in Equation 2 is a stochastic process modeled by a matching solver (M). Conventional methods calculate a single distance (D e) is calculated, and a deterministic algorithm is used for the matching solver. Therefore, the optimization problems of Equation 1 and Equation 2 are solved independently. As a result, it may be difficult to improve the accuracy of matching.

[0064] In contrast, in this embodiment, the inference of distance D is extended to non-deterministic inference (distribution φ), and the matching model 50 is configured to accept input of the probability distribution 40 (φ) of the distance between each sample (30, 35). This makes it possible for the operation (m) of the matching model 50 to be connected to the operation (ε) of feature extraction. In other words, the calculation process of the matching task according to this embodiment can be expressed by the following equation 3.

[0065]

number

[0066] Therefore, in this embodiment, the two probabilistic processes of inference by the feature extractor 60 and inference (matching) by the matching model 50 can be computationally modeled as a single probabilistic process. Additionally, since the feature extractor 60 is configured using differentiable operations, performing machine learning by the feature extractor 60 can be considered as solving a single optimization problem expressed by Equation 3. Solving the single optimization problem expressed by Equation 3 is an example of machine learning by the feature extractor 60 in cooperation with the matching model 50. In this machine learning, the target of optimization may be the feature extractor 60 alone, or both the feature extractor 60 and the matching model 50. In the latter case, the matching model 50 may be configured using a machine learning model and trained by machine learning integrated with the feature extractor 60. In either case, compared to performing machine learning by the feature extractor 60 separately, taking into account the operations of the matching model 50 may potentially result in a better optimal solution for the matching task by the feature extractor 60. Therefore, according to this example of the present embodiment, the machine learning device 2 can be expected to generate a feature extractor 60 that is more suited to the matching task. In the matching device 1, by using a trained feature extractor 60 generated by such machine learning for a matching task, it is possible to expect an improvement in matching accuracy.

[0067] In one example, when the matching model 50 is configured by a machine learning model as described above, training the feature extractor 60 may be configured by jointly training the feature extractor 60 and the matching model 50. The trained matching model 50 may be provided to the matching device 1 from the machine learning device 2 in any manner and at any timing, similar to the feature extractor 60. In response, the matching device 1 may use the matching model 50 trained by machine learning for the matching task. That is, in the matching device 1, the matching model 50 may be a machine learning model trained by machine learning jointly with the feature extractor 60. Joint machine learning may mean that, as shown in Equation 3 above, a series of operations from the feature extractor 60 to the matching model 50 can be expressed by a single computational model, and the computational parameters of the feature extractor 60 and the matching model 50 expressed by a single computational model are optimized.

[0068] In conventional methods, the two optimization problems of Equation 1 and Equation 2 are solved independently. This can lead to local optima in the feature extractor and the matching solver, respectively, making it difficult to improve the accuracy of matching. In contrast, according to an example of the present embodiment, by integrally executing machine learning for the feature extractor 60 and the matching model 50, it is possible to obtain better optimal solutions for the matching task for the feature extractor 60 and the matching model 50. That is, the feature extractor 60 can be trained to acquire the ability to calculate features suitable for matching (inference) by the matching model 50. The matching model 50 can be trained to acquire the ability to perform matching (inference) processing suitable for the features calculated by the feature extractor 60. Therefore, the machine learning device 2 can be expected to generate a feature extractor 60 and a matching model 50 that are more suitable for the matching task. In the matching device 1, by using the trained feature extractor 60 and the matching model 50 generated by such machine learning for the matching task, it can be expected to improve the accuracy of matching. Note that the machine learning method of the feature extractor 60 is not limited to these examples and may be changed as appropriate depending on the embodiment. In another example, the machine learning of the feature extractor 60 may be performed separately.

[0069] (Selected) In one example of this embodiment, the above-described selection configuration that can be employed in the inference (matching) stage may also be employed in the learning stage. That is, calculating a distance probability distribution 45 from the feature quantities (34, 39) of each element may be configured by: calculating an evaluation value between each element of the first training sample 33 and the second training sample 38 from the calculated feature quantities (34, 39); selecting one or more combinations of elements from the combinations of elements of the first training sample 33 and the second training sample 38 based on the calculated evaluation value; and calculating a probability distribution 45 of the distance between the elements of each of the selected one or more combinations from the calculated feature quantities (34, 39). Accordingly, the probability distribution 45 of the distance between the first training sample 33 and the second training sample 38 may be configured from the probability distribution 45 of the distance between the elements of each of the selected one or more combinations. According to this example of this embodiment, the processing load (computational cost) of the matching model 50 can be reduced.

[0070] (summary) In addition, in one example of this embodiment, a configuration for summarizing the above feature quantities that can be employed in the inference stage may also be employed in the learning stage. That is, a probability distribution 45 of the distance between elements may be calculated from a portion of the feature quantities (34, 39) of each element. By summarizing the feature quantities (34, 39) of each element, the feature quantities (34, 39) used to calculate the probability distribution 45 are narrowed down to a portion. As a result, the amount of data in the probability distribution 45 can be reduced. Therefore, by summarizing the feature quantities (34, 39) of each element, the processing load (computational cost) in the matching model 50 can also be reduced.

[0071] (Controlling Machine Learning) Furthermore, controlling machine learning may include executing machine learning processing within the machine learning device 2, and giving instructions to an external computer other than the machine learning device 2 to have the external computer execute the machine learning processing. That is, in one example, the machine learning device 2 may execute the machine learning processing. In another example, the machine learning device 2 may give an instruction to execute the machine learning processing to an external computer. The external computer may execute the machine learning processing in response to an instruction from the machine learning device 2. In yet another example, the machine learning device 2 may execute part of the machine learning processing, and the external computer may execute the rest of the machine learning processing. The external computer may be composed of one or more computers. For example, the external computer is a server device. The external computer may be connected to the machine learning device 2 via a network, or may be directly connected to the machine learning device 2. The type of network is not particularly limited, and may be any type of network in the embodiments. It may be selected appropriately depending on the situation.

[0072] (Outputs the results of machine learning) Outputting the machine learning result may include outputting any information related to the machine learning result. In one example, outputting the machine learning result may include generating learning result data indicating the machine learning result and storing the generated learning result data in any storage area. In another example, outputting the machine learning result may include outputting a convergence result of training in machine learning. The convergence result may be indicated by, for example, a loss, a number of time steps, a learning curve, etc.

[0073] (System Configuration) In one example, as shown in Figure 5, the matching device 1 and the machine learning device 2 may be connected to each other via a network. The type of network may be appropriately selected from, for example, the Internet, a wireless communication network, a mobile communication network, a telephone network, a dedicated network, etc. However, the method of exchanging data between the matching device 1 and the machine learning device 2 is not limited to this example and may be appropriately selected depending on the embodiment. In another example, data may be exchanged using a storage medium.

[0074] 5, the matching device 1 and the machine learning device 2 are configured as separate computers. However, the configuration of the system according to this embodiment is not limited to this example and may be determined appropriately depending on the embodiment. In another example, the matching device 1 and the machine learning device 2 may be configured as a single computer. In yet another example, at least one of the matching device 1 and the machine learning device 2 may be configured as multiple computers.

[0075] Matching Task The matching task may include any task of inferring combinations of elements among multiple samples, each of which includes multiple elements. In other words, this embodiment is applicable to any combinatorial optimization problem. Each sample (input) is a set of elements that are the target of the matching task. As long as the elements can be the target of the matching task, the configuration of the elements is not particularly limited and may be determined appropriately depending on the embodiment. The elements may be expressed by any numerical values, such as points, pixels, or attribute values. The method of acquiring each sample may be selected appropriately depending on the embodiment. In one example, each sample may be obtained by a sensor or by computer information processing (data augmentation, simulation, etc.). The process of acquiring each sample may at least partially involve manual operation by an operator.

[0076] <Scene example> The system according to this embodiment may be applied to various situations where any matching task is performed. The application of this embodiment may include at least one of the following first, second, third, fourth, and fifth cases. Specific application situations are exemplified below for each case.

[0077] (1) First Case 6 is a diagram showing an example of a first case to which this embodiment is applied. The first case is an example of a scene in which this embodiment is applied to matching between point cloud data.

[0078] 6, in the learning stage and the inference (matching) stage, the first input may be the first point cloud data D1, and the second input may be the second point cloud data D2. Accordingly, the matching between elements may be the matching between points included in the first point cloud data D1 and points included in the second point cloud data D2. The points included in each point cloud data (D1, D2) are an example of elements included in each input.

[0079] In the learning stage, the method for acquiring the training samples (33, 38) of each point cloud data (D1, D2) is not particularly limited and may be appropriately selected depending on the embodiment. The training samples (33, 38) may be acquired using any sensor, such as a LiDAR (light detection and ranging), a MMS (Mobile Mapping System), a depth sensor, an ultrasonic sensor, an infrared sensor, or a radar. The same applies to the acquisition of the samples (30, 35) in the inference stage.

[0080] According to the first example, in the matching device 1, improvement in matching accuracy can be expected when matching points between each piece of point cloud data (D1, D2). When a configuration for performing machine learning of the feature extractor 60 is adopted, it can be expected that the machine learning device 2 will generate a feature extractor 60 that is suitable for matching between the point cloud data (D1, D2). When a configuration for integrally executing machine learning of the feature extractor 60 and the matching model 50 is adopted, it can be expected that the feature extractor 60 and the matching model 50 that are suitable for matching between the point cloud data (D1, D2) will be generated.

[0081] (2) Second Case 7 is a diagram showing an example of a second case to which this embodiment is applied. The second case is an example of a scene in which this embodiment is applied to matching between images.

[0082] 7, in the learning stage and the inference stage, the first input may be a first image D3, and the second input may be a second image D4. Accordingly, the matching between elements may be the matching between points included in the first image D3 and points included in the second image D4. The points in each image (D3, D4) may be, for example, pixels, feature points, etc. The points included in each image (D3, D4) are examples of elements included in each input.

[0083] Matching of points between images may be performed for any purpose. For example, matching of points between images may be performed by matching a combination of feature points between images taken at different times in order to generate optical flow. Images taken at different times are an example of each image (D3, D4). Also, for example, matching of points between images may be performed by matching a combination of feature points between images obtained by each of multiple cameras (such as a stereo camera) in order to calculate parallax. Images obtained by each camera are an example of each image (D3, D4).

[0084] In the learning stage, the method for acquiring the training samples (33, 38) of the images (D3, D4) is not particularly limited and may be appropriately selected depending on the embodiment. For example, an image sensor such as an RGB camera may be used to acquire the training samples (33, 38). The same applies to the acquisition of the samples (30, 35) in the inference stage.

[0085] According to the second example, in the matching device 1, improvement in matching accuracy can be expected when matching points between images (D3, D4). When a configuration for performing machine learning of the feature extractor 60 is adopted, it can be expected that the machine learning device 2 will generate a feature extractor 60 suitable for matching between images (D3, D4). When a configuration for integrally executing machine learning of the feature extractor 60 and the matching model 50 is adopted, it can be expected that the feature extractor 60 and the matching model 50 suitable for matching between images (D3, D4) will be generated.

[0086] (3) Third Case FIG. 8 schematically shows an example of a third case to which this embodiment is applied. The third case is an example of a situation in which this embodiment is applied to person-to-person matching. A person may include a specific individual, an unspecified person, and an organization. A person may be an actual entity or a virtual entity.

[0087] As shown in FIG. 8, in the learning stage and the inference stage, the first input may be a first person set D5, and the second input may be a second person set D6. Accordingly, matching between elements may be matching between people included in the first person set D5 and people included in the second person set D6. A person included in the first person set D5 may be referred to as a "first person," and a person included in the second person set D6 may be referred to as a "second person." The people included in each person set (D5, D6) are examples of elements included in each input. A person (element) may be appropriately represented, for example, by an attribute value or the like.

[0088] Matching of people between groups of people may be performed for any purpose. The matching target may be, for example, donor / client, doctor / patient, male / female, roommate, job seeker / employer, employee / placement, energy supplier / receiver, game player, etc. As an example, this embodiment may be applied to matchmaking players who meet conditions for exciting a match (such as similar skills).

[0089] The method for acquiring the training samples (33, 38) for each person set (D5, D6) during the learning phase is not particularly limited and may be selected appropriately depending on the embodiment. In one example, the training samples (33, 38) may be acquired at least partially by manual operation. The training samples (33, 38) may be acquired from actual people or from virtual people (such as simulations). The same applies to the acquisition of the samples (30, 35) during the inference phase.

[0090] According to the third example, in the matching device 1, improvement in matching accuracy can be expected when matching between people in each person set (D5, D6). When a configuration for performing machine learning of the feature extractor 60 is adopted, it can be expected that the machine learning device 2 will generate a feature extractor 60 suitable for matching between people sets (D5, D6). When a configuration for integrally executing machine learning of the feature extractor 60 and the matching model 50 is adopted, it can be expected that the feature extractor 60 and the matching model 50 suitable for matching between people sets (D5, D6) will be generated.

[0091] (4) Fourth Case FIG. 9 schematically shows an example of a fourth case to which this embodiment is applied. The fourth case is an example of a situation in which this embodiment is applied to person-object matching. The object may be any object that is limited in number (i.e., finite). The object may be a tangible object or an intangible object (such as a service). The object may be an actual entity or a virtual entity. A person may be defined in the same way as in the third case.

[0092] 9, in the learning stage and the inference stage, the first input may be a person set D7, and the second input may be an entity set D8. Accordingly, the matching between elements may be a matching between the people included in the person set D7 and the entities included in the entity set D8. The people included in the person set D7 and the entities included in the entity set D8 are each an example of an element included in each input. Similar to people, entities (elements) may also be appropriately represented by, for example, attribute values.

[0093] Matching of people and objects between a set of people and a set of objects may be performed for any purpose, such as people / inventory, users / web advertisements with a limited number of submissions, guests / hotel rooms (automatic check-in at accommodations), purchasers / seats (e.g., seat allocation in public transportation, entertainment facilities, etc.), students / classes, applicants / jobs, etc.

[0094] In the learning stage, the method of acquiring the training samples (33, 38) of the person set D7 and the object set D8 may not be particularly limited and may be appropriately selected depending on the embodiment. In one example, each training sample (33, 38) may be acquired at least partially by manual operation. Each training sample (33, 38) may be acquired from a real entity or from a virtual entity. The same applies to obtaining each sample (30, 35) in the inference stage.

[0095] According to the fourth example, in the matching device 1, improvement in matching accuracy can be expected when performing person-object matching between a person set D7 and an object set D8. When a configuration for performing machine learning of the feature extractor 60 is adopted, it can be expected that the machine learning device 2 will generate a feature extractor 60 that is suitable for matching between a person set D7 and an object set D8. When a configuration for integrally executing machine learning of the feature extractor 60 and the matching model 50 is adopted, it can be expected that the feature extractor 60 and the matching model 50 that are suitable for matching between a person set D7 and an object set D8 will be generated.

[0096] (5) Fifth Case 10 is a schematic diagram illustrating an example of a fifth case to which this embodiment is applied. The fifth case is an example of a scene in which this embodiment is applied to object-object matching. The object may be defined in the same way as in the fourth case.

[0097] 10, in the learning stage and the inference stage, the first input may be a first object set D9, and the second input may be a second object set D10. Accordingly, the matching between elements may be a matching between an object included in the first object set D9 and an object included in the second object set D10. An object included in the first object set D9 may be referred to as a "first object," and an object included in the second object set D10 may be referred to as a "second object." The objects included in each object set (D9, D10) are examples of elements included in each input.

[0098] Matching between objects between object sets may be performed for any purpose. The target of matching may be, for example, a combination of parts, materials / factories, products / delivery destinations, data / storage areas, transport robots / baggage, etc. For example, this embodiment may be applied to combine parts in a tray so as to minimize the defect rate. Furthermore, this embodiment may be applied to combine material warehouses and factories so as to shorten the delivery distance. Similarly, this embodiment may be applied to combine product warehouses and delivery destinations so as to shorten the delivery distance of the products.

[0099] In the learning stage, the method for acquiring the training samples (33, 38) for each object set (D9, D10) is not particularly limited and may be selected appropriately depending on the embodiment. In one example, the training samples (33, 38) may be acquired at least partially by manual operation. The training samples (33, 38) may be acquired from real entities or from virtual entities (such as simulations). The same applies to the acquisition of the samples (30, 35) in the inference stage.

[0100] According to the fifth example, in the matching device 1, improvement in matching accuracy can be expected when matching between entities between each entity set (D9, D10). When a configuration for performing machine learning of the feature extractor 60 is adopted, it can be expected that the machine learning device 2 will generate a feature extractor 60 that is suitable for matching between entity sets (D9, D10). When a configuration for integrally executing machine learning of the feature extractor 60 and the matching model 50 is adopted, it can be expected that the feature extractor 60 and the matching model 50 that are suitable for matching between entity sets (D9, D10) will be generated.

[0101] §2 Configuration example [Hardware configuration] (Verification device) 11 is a schematic diagram illustrating an example of the hardware configuration of the verification device 1 according to this embodiment. The verification device 1 according to this embodiment includes a control unit 11, a storage unit 12, a communication interface 13, An input device 14, an output device 15, and a drive 16 are electrically connected to the computer.

[0102] The control unit 11 includes a CPU (Central Processing Unit) which is a hardware processor, The memory unit 12 includes RAM (Random Access Memory), ROM (Read Only Memory), etc., and is configured to execute information processing based on programs and various data. The control unit 11 (CPU) is an example of a processor resource. The storage unit 12 may be configured, for example, with a hard disk drive, a solid state drive, etc. The storage unit 12, RAM, and ROM are examples of memory resources. In this embodiment, the storage unit 12 stores various information such as a matching program 81, matching model data 500, feature extractor data 600, etc.

[0103] The matching program 81 is a program for causing the matching device 1 to execute information processing (see FIG. 16 described below) related to the performance of a matching task. The matching program 81 includes a series of instructions for the information processing. The matching model data 500 is configured to indicate information related to the matching model 50. The feature extractor data 600 is configured to indicate information related to the feature extractor 60. If the matching model 50 is generated based on machine learning, the matching model data 500 is an example of learning result data for the matching model 50. If the feature extractor 60 is generated based on machine learning, the feature extractor data 600 is an example of learning result data for the feature extractor 60. If the feature extractor 60 is not used, the feature extractor data 600 may be omitted.

[0104] As long as the configuration of the matching model data 500 can hold information for executing the calculation processing of the matching model 50, the configuration of the matching model data 500 is not particularly limited and may be determined appropriately depending on the embodiment. For example, if the matching model 50 is configured using a machine learning model, the matching model data 500 may be configured to include information indicating values ​​of calculation parameters of the matching model 50 adjusted by machine learning. The matching model data 500 may also be configured to further include information indicating the configuration of the matching model 50 (for example, the structure of a neural network, etc.). The same applies to the feature extractor data 600. For example, at least one of the matching model data 500 and the feature extractor data 600 may be incorporated into the matching program 81.

[0105] The communication interface 13 is configured to perform wired or wireless communication via a network. The communication interface 13 may be configured, for example, by a wired LAN (Local Area Network) module, a wireless LAN module, etc. The collation device 1 may perform data communication with another computer (for example, the machine learning device 2, etc.) via the communication interface 13.

[0106] The input device 14 is a device for inputting, for example, a mouse, a keyboard, etc. The output device 15 is a device for outputting, for example, a display, a speaker, etc. An operator can operate the verification device 1 by using the input device 14 and the output device 15. The input device 14 and the output device 15 may be connected via an external interface. The external interface may be appropriately configured to connect to an external device by wire or wirelessly, for example, via a USB (Universal Serial Bus) port, a dedicated port, etc. The input device 14 and the output device 15 may be integrated into one device, for example, a touch panel display, etc.

[0107] The drive 16 is a device for reading various information such as programs stored in the storage medium 91. At least one of the above-mentioned matching program 81, matching model data 500, and feature extractor data 600 may be stored in the storage medium 91 instead of or together with the storage unit 12. The storage medium 91 is configured to store various information (stored programs, etc.) by electrical, magnetic, optical, mechanical, or chemical action so that a machine such as a computer can read the information. The storage unit 12 and the storage medium 91 are non-uniform. The storage medium 91 is an example of a temporary storage medium. The matching device 1 may acquire at least one of the matching program 81, the matching model data 500, and the feature extractor data 600 from the storage medium 91. The storage medium 91 may be a disk-type storage medium such as a CD or a DVD, or may be a non-disk-type storage medium such as a semiconductor memory (e.g., a flash memory). The type of the drive 16 may be selected appropriately depending on the type of the storage medium 91. The drive 16 may be connected via an external interface.

[0108] Note that, with regard to the specific hardware configuration of the matching device 1, components may be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit 11 may include multiple hardware processors. The hardware processor may be configured with a microprocessor, a field-programmable gate array (FPGA), a digital signal processor (DSP), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or the like. At least one of the communication interface 13, the input device 14, the output device 15, and the drive 16 may be omitted. The matching device 1 may be connected to a sensor via the communication interface 13 or an external interface. The matching device 1 may acquire each input sample (30, 35) from the sensor. At least one of the matching program 81, the matching model data 500, and the feature extractor data 600 may be stored in an external storage device such as a network-attached storage (NAS). The matching device 1 may be configured with multiple computers. In this case, the hardware configurations of the computers may or may not be identical. Furthermore, the verification device 1 may be an information processing device designed specifically for the service provided, or may be a general-purpose server device, a general-purpose PC (Personal Computer), a tablet PC, a terminal device, or the like.

[0109] (machine learning device) 12 schematically illustrates an example of the hardware configuration of a machine learning device 2 according to this embodiment. The machine learning device 2 according to this embodiment is a computer in which a control unit 21, a storage unit 22, a communication interface 23, an input device 24, an output device 25, and a drive 26 are electrically connected.

[0110] The control unit 21 to the drive 26 and the storage medium 92 of the machine learning device 2 may be configured similarly to the control unit 11 to the drive 16 and the storage medium 91 of the above-described matching device 1, respectively. The control unit 21 (CPU) is an example of a processor resource of the machine learning device 2. The storage unit 22 (and RAM, ROM) is an example of a memory resource of the machine learning device 2. In this embodiment, the storage unit 22 stores various information such as a machine learning program 82, matching model data 500, feature extractor data 600, and multiple data sets 700.

[0111] The machine learning program 82 is a program for causing the machine learning device 2 to execute information processing (see FIG. 15 described below) related to the machine learning of the feature extractor 60. The machine learning program 82 includes a series of instructions for this information processing. In one example, the matching model data 500 and the feature extractor data 600 may be generated as a result of executing the machine learning program 82. If the machine learning device 2 does not generate a trained matching model 50, the matching model data 500 may be omitted. Each dataset 700 is used for machine learning in the feature extractor 60. The configuration of each dataset 700 may be appropriately determined so that it can be used for machine learning in the feature extractor 60. In one example, each dataset 700 may be composed of a combination of a first training sample 33, a second training sample 38, and a true value 70.

[0112] At least one of the machine learning program 82, the matching model data 500, the feature extractor data 600, and the plurality of data sets 700 may be stored in a storage medium 92 instead of or together with the storage unit 22. The machine learning device 2 stores at least one of the machine learning program 82, the matching model data 500, the feature extractor data 600, and the plurality of data sets 700. At least one of the machine learning program 82, the matching model data 500, the feature extractor data 600, and the plurality of data sets 700 may be acquired from the storage medium 92. At least one of the machine learning program 82, the matching model data 500, the feature extractor data 600, and the plurality of data sets 700 may be stored in an external storage device.

[0113] The machine learning device 2 may communicate data with other computers (e.g., the collation device 1) via the communication interface 23. An operator can operate the machine learning device 2 by using the input device 24 and the output device 25.

[0114] Note that with regard to the specific hardware configuration of the machine learning device 2, components can be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit 21 may include multiple hardware processors. The hardware processor may be configured with a microprocessor, FPGA, DSP, GPU, ASIC, etc. At least one of the communication interface 23, input device 24, output device 25, and drive 26 may be omitted. The machine learning device 2 may be configured with multiple computers. In this case, the hardware configurations of the computers may or may not be the same. The machine learning device 2 may be an information processing device designed specifically for the service provided, as well as a general-purpose server device, a general-purpose PC, a tablet PC, a terminal device, etc.

[0115] [Software configuration] (Verification device) 13 schematically shows an example of the software configuration of the collation device 1 according to this embodiment. The control unit 11 of the collation device 1 executes instructions included in the collation program 81 stored in the storage unit 12 using the CPU. As a result, the collation device 1 operates as a computer including an acquisition unit 111, a collation unit 112, and an output processing unit 113 as software modules. That is, in this embodiment, each software module of the collation device 1 is realized by the control unit 11 (CPU).

[0116] The acquisition unit 111 is configured to acquire a probability distribution 40 of a distance between a first sample 30 of a first input and a second sample 35 of a second input. In one example, the acquisition unit 111 may include a feature extractor 60 by holding feature extractor data 600. The acquisition unit 111 may be configured to acquire the probability distribution 40 of a distance between each sample (30, 35) using the feature extractor 60. The matching unit 112 includes a matching model 50 by holding matching model data 500. The matching unit 112 is configured to provide the acquired probability distribution 40 to the matching model 50 and execute calculation processing of the matching model 50, thereby acquiring a matching result between elements of the first sample 30 and the second sample 35 from the matching model 50. The output processing unit 113 is configured to output information related to the acquired matching result.

[0117] (machine learning device) 14 schematically shows an example of the software configuration of the machine learning device 2 according to this embodiment. The control unit 21 of the machine learning device 2 executes instructions included in the machine learning program 82 stored in the storage unit 22 using the CPU. As a result, the machine learning device 2 operates as a computer including an acquisition unit 211, a learning processing unit 212, and an output processing unit 213 as software modules. That is, in this embodiment, similar to the collation device 1, each software module of the machine learning device 2 is also realized by the control unit 21 (CPU).

[0118] The acquisition unit 211 is configured to acquire each training sample (33, 38). In one example, the acquisition unit 211 may be configured to acquire a plurality of data sets 700. The learning processing unit 212 is configured to control the machine learning of the feature extractor 60 using each acquired training sample (33, 38). In one example, the learning processing unit 212 may be configured to control the machine learning of the feature extractor 60 in cooperation with the matching model 50. 3 is configured to output the results of the machine learning.

[0119] (others) In this embodiment, an example is described in which each software module of the matching device 1 and the machine learning device 2 is implemented by a general-purpose CPU. However, some or all of the above software modules may be implemented by one or more dedicated processors or chipsets. Each of the above modules may be implemented as a hardware module. With regard to the software configuration of the matching device 1 and the machine learning device 2, modules may be omitted, replaced, or added as appropriate depending on the embodiment.

[0120] §3 Example of operation [Machine learning device] FIG. 15 is a flowchart showing an example of the processing procedure of the machine learning device 2 according to this embodiment. The following processing procedure is an example of a machine learning method (model generation method) executed by a computer. However, the following processing procedure of the machine learning device 2 is merely an example, and each step may be modified as much as possible. Furthermore, steps in the following processing procedure may be omitted, replaced, or added as appropriate depending on the embodiment.

[0121] (Step S101) In step S101, the control unit 21 operates as the acquisition unit 211 and acquires each training sample (33, 38). In one example, the control unit 21 may acquire a plurality of data sets 700. Each data set 700 may be composed of a combination of a first training sample 33, a second training sample 38, and a true value 70. The data type of each training sample (33, 38) may be selected from any of the first to fifth cases described above.

[0122] When machine learning is performed in an external computer, the control unit 21 may acquire at least a portion of each training sample (33, 38) and provide the acquired training samples (33, 38) to the external computer. Alternatively, acquiring each training sample (33, 38) may include giving an instruction to the external computer to cause the external computer to acquire at least a portion of each training sample (33, 38). After acquiring each training sample (33, 38), the control unit 21 proceeds to the next step S102.

[0123] (Step S102) In step S102, the control unit 21 operates as the learning processing unit 212 and controls the machine learning of the feature extractor 60. The acquired training samples (33, 38) are used in the machine learning of the feature extractor 60. The number of training samples (33, 38) used in the machine learning is not particularly limited and may be determined appropriately depending on the embodiment.

[0124] In one example, the control unit 21 may control machine learning of the feature extractor 60 in cooperation with the matching model 50. The machine learning process may include using the feature extractor 60 to calculate feature quantities (34, 39) of each element included in each training sample (33, 38), calculating a probability distribution 45 of the distance between each training sample (33, 38) from the calculated feature quantities (34, 39) of each element, providing the calculated probability distribution 45 of the distance to the matching model 50 and executing calculation processing of the matching model 50 to derive a matching result between the elements of each training sample (33, 38), and training the feature extractor 60 so that the derived matching result approaches the corresponding true value 70.

[0125] In one example, calculating the probability distribution 45 of distances from the feature values ​​(34, 39) of each element includes calculating an evaluation value between elements of each training sample (33, 38) from the calculated feature values ​​(34, 39), and calculating the probability distribution 45 of distances from the feature values ​​(34, 39) of each element. The method may be configured by selecting one or more combinations from the combinations, and calculating a probability distribution 45 of the distances between the elements of each of the selected one or more combinations from the calculated feature quantities (34, 39).

[0126] In one example, the control unit 21 may execute the machine learning of the feature extractor 60 within the machine learning device 2. In another example, the control unit 21 may issue an instruction to an external computer to execute the machine learning of the feature extractor 60. The external computer may execute the machine learning of the feature extractor 60 in response to an instruction from the machine learning device 2.

[0127] As a result of performing machine learning, a trained feature extractor 60 can be generated that has acquired the ability to calculate features within the category of each training sample (33, 38) used. By selecting any of the first to fifth cases as the data type of each training sample (33, 38), a trained feature extractor 60 that can be used for matching tasks of any of the cases can be generated.

[0128] In one example, the matching model 50 may be configured by a machine learning model, and the machine learning of the feature extractor 60 may be executed integrally with the machine learning of the matching model 50. In this case, a trained matching model 50 can be generated together with the trained feature extractor 60. The machine learning process of the matching model 50, like the feature extractor 60, may be executed within the machine learning device 2 or may be executed on an external computer. By selecting any of the first to fifth cases as the data type of each training sample (33, 38), it is possible to generate a matching model 50 that can be used for the matching task of any of the cases. When the machine learning process is completed, the control unit 21 proceeds to the next step S103.

[0129] (Step S103) In step S103, the control unit 21 operates as the output processing unit 213 to output the results of the machine learning. The output destination and the content of the information to be output may be selected appropriately depending on the embodiment.

[0130] In one example, the control unit 21 may generate feature extractor data 600 (learning result data) indicating the trained feature extractor 60 as output processing and store the generated feature extractor data 600 in a predetermined storage area. In this case, the feature extractor data 600 is an example of output information related to the results of machine learning. The predetermined storage area may be, for example, RAM in the control unit 21, the storage unit 22, an external storage device, a storage medium, or a combination thereof. The storage medium may be, for example, a CD, a DVD, a semiconductor memory, or the like. The external storage device may be, for example, a data server such as a NAS. The external storage device may be, for example, an external storage device. When the machine learning calculation process is performed on an external computer, the feature extractor data 600 may be generated by the external computer. When a configuration is adopted in which machine learning of the matching model 50 is performed together with the feature extractor 60, the same applies to the matching model data 500. The control unit 21 may store the matching model data 500 (learning result data) in a predetermined storage area. The matching model data 500 may be generated by an external computer. In another example, the control unit 21 may output a convergence result of machine learning training as the output process. The output destination may be, for example, RAM in the control unit 21, the output device 25, an external computer, an external storage device, a storage medium, or a combination thereof. In yet another example, the control unit 21 may output the progress of machine learning training as the output process. In this case, the process of step S103 may be executed together with step S102.

[0131] When the output of the machine learning results is completed, the control unit 21 ends the processing procedure of the machine learning device 2 according to this operation example.

[0132] Note that when the matching device 1 uses the trained feature extractor 60, the generated feature extractor data 600 may be provided to the matching device 1 from the machine learning device 2 or an external computer at any time and by any method. In one example, the feature extractor data 600 may be provided to the matching device 1 via a network, a storage medium, or the like. In another example, the feature extractor data 600 may be pre-installed in the matching device 1. The same applies to the matching model data 500. When a configuration is adopted in which machine learning of the matching model 50 is performed together with the feature extractor 60, the matching model data 500 may be provided to the matching device 1 from the machine learning device 2 or an external computer at any time and by any method.

[0133] Furthermore, the control unit 21 may update or generate new feature extractor data 600 by periodically or irregularly repeating the processes of steps S101 to S103. During this repetition, at least a portion of the training samples (33, 38) may be changed, modified, added, deleted, or the like as appropriate. The control unit 21 may then provide the updated or newly generated feature extractor data 600 to the matching device 1 as appropriate. As a result, the feature extractor data 600 held in the matching device 1 may be updated. The same applies to the matching model data 500. The control unit 21 may update or generate new matching model data 500 by periodically or irregularly repeating the processes of steps S101 to S103. The control unit 21 may provide the updated or newly generated matching model data 500 to the matching device 1 as appropriate. As a result, the matching model data 500 held in the matching device 1 may be updated.

[0134] [Verification device] FIG. 16 is a flowchart showing an example of the processing procedure of the verification device 1 according to this embodiment. The following processing procedure is an example of a verification method executed by a computer. However, the following processing procedure of the verification device 1 is merely an example, and each step may be changed as much as possible. Furthermore, steps in the following processing procedure may be omitted, replaced, or added as appropriate depending on the embodiment.

[0135] (Step S201) In step S201, the control unit 11 operates as the acquisition unit 111 and acquires the probability distribution 40 of the distance between each sample (30, 35).

[0136] In one example, obtaining the distance probability distribution 40 may be configured by using a feature extractor 60 to calculate feature quantities (31, 36) of each element included in each sample (30, 35), and calculating the distance probability distribution 40 between each sample (30, 35) from the calculated feature quantities (31, 36) of each element. The process of calculating the probability distribution 40 using the feature extractor 60 may be executed within the matching device 1 or may be executed by an external computer other than the matching device 1. In the latter case, the control unit 11 may obtain the calculated probability distribution 40 from the external computer. The matching model 50 may be configured using differentiable operations, and the feature extractor 60 may be trained by machine learning in conjunction with the matching model 50.

[0137] Furthermore, calculating the distance probability distribution 40 from the feature quantities (31, 36) of each element may be configured by calculating an evaluation value between each element of each sample (30, 35) from the calculated feature quantities (31, 36), selecting one or more combinations from the combinations of elements of each sample (30, 35) based on the calculated evaluation values, and calculating the distance probability distribution 40 between the elements of each of the selected one or more combinations from the feature quantities (31, 36). In another example, the distance probability distribution 40 may be obtained by a method other than the method using the feature extractor 60 (such as the above-mentioned multiple observation method).

[0138] In one example, the data type of each sample (30, 35) may be selected from any of the first to fifth cases. If so, the control unit 11 advances the process to the next step S202.

[0139] (Step S202) In step S202, the control unit 11 operates as the matching unit 112, provides the acquired probability distribution 40 to the matching model 50, and executes the calculation processing of the matching model 50 to obtain the matching results between the elements of each sample (30, 35) from the matching model 50.

[0140] The arithmetic processing of the matching model 50 may be defined as appropriate depending on the embodiment, such as the configuration of the matching model 50. For example, if the matching model 50 is configured by a neural network, the control unit 11 may execute forward arithmetic processing of the matching model 50. As a result of executing this arithmetic processing, the control unit 11 can obtain an output corresponding to the matching result from the matching model 50.

[0141] In one example, the matching model 50 may be a machine learning model trained by machine learning integrated with the feature extractor 60. When any of the first to fifth cases is selected as the data type of each sample (30, 35), a matching result corresponding to any of the cases can be obtained. Upon obtaining the matching result, the control unit 11 proceeds to the next step S203.

[0142] (Step S203) In step S203, the control unit 11 operates as the output processing unit 113 and outputs information relating to the acquired matching result.

[0143] The output destination and the content of the information to be output may be selected appropriately depending on the embodiment. In one example, the control unit 11 may output the acquired matching result as is. In another example, the control unit 11 may execute any information processing depending on the acquired matching result. The control unit 11 may output the result of the information processing as information related to the matching result. The output of the result of the information processing may include, for example, outputting a specific message depending on the matching result, or controlling the operation of the controlled device depending on the matching result. The output destination may be, for example, RAM, the memory unit 12, the output device 15, another computer, the controlled device, etc. For example, in the scene of the fifth example above, when the transport robot and the luggage are matched, the control unit 11 may instruct the transport robot to move toward the paired luggage. The transport robot is an example of the controlled device, and the movement instruction is an example of operation control.

[0144] When the output of information is completed, the control unit 11 ends the processing procedure of the verification device 1 according to this operation example. The control unit 11 may execute a series of processes from step S201 to step S203 at any timing such as a user operation, satisfaction of a condition, etc. The control unit 11 may execute the series of processes from step S201 to step S203 in real time, or may execute them as a post-facto verification process.

[0145] [Features] In this embodiment, in step S202, a matching model 50 configured to derive a matching result between elements contained in each input from the probability distribution of the distance between each input is used for the matching task. This makes it possible to infer the combination of elements between samples (30, 35) while taking into account the possibility (uncertainty) that the given distance is incorrect. As a result, it is expected that matching accuracy will be ensured even in situations where the accuracy of the given distance is low. Therefore, according to this embodiment, it is expected that the accuracy of matching between elements will be improved.

[0146] Furthermore, in this embodiment, in step S102, machine learning of the feature extractor 60 is performed in conjunction with the calculation of the matching model 50, which is expected to generate a feature extractor 60 that is more suited to the matching task. By using the generated trained feature extractor 60 in step S201, it is expected that the accuracy of matching will be improved.

[0147] Furthermore, in this embodiment, in step S102, machine learning of the matching model 50 may be executed integrally with machine learning of the feature extractor 60. This can be expected to generate a feature extractor 60 and a matching model 50 that are more suited to the matching task. By using the trained feature extractor 60 generated by this integrated machine learning in step S201 and the trained matching model 50 in step S202, it can be expected that the accuracy of matching will be improved.

[0148] §4 Variations Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. The processes and means described in this disclosure can be freely combined and implemented as long as no technical contradiction occurs. Furthermore, various improvements or modifications may be made to the above embodiments as appropriate.

[0149] §5 Working Examples The following experiments were carried out to verify the effectiveness of the above-described embodiment, but the present invention is not limited to the following examples.

[0150] [First Experimental Example] First, as a preliminary step, four datasets were prepared: the first dataset (4DMatch), the second dataset (4DLoMatch), the third dataset (3DMatch), and the fourth dataset (3DLoMatch).

[0151] The first dataset (4DMatch) and the second dataset (4DLoMatch) were proposed in Non-Patent Document 1. Point cloud data of 1761 animals (non-rigid bodies) were acquired for each of the first dataset (4DMatch) and the second dataset (4DLoMatch). Of the 1761 point cloud data, 1232 were used for training, 176 for validation, and 353 for testing.

[0152] The third dataset (3DMatch) is proposed in Non-Patent Document 3. The third dataset (3DMatch) and the fourth dataset (3DLoMatch) were each made up of 62 indoor (rigid) point cloud data sets. Of the 62 point cloud data sets, 46 were used for training, 8 for validation, and 8 for testing.

[0153] (Comparative Example) In the first comparative example, the calculations from feature extraction of each input to matching were configured using the method (Lepard) proposed in Non-Patent Document 1. In the second comparative example, the method (LNDP) proposed in Non-Patent Document 5 was adopted. In the third comparative example, the method (RoITr) proposed in Non-Patent Document 6 was adopted. In the first and second comparative examples, dual softmax was used as the matching solver. In the third comparative example, the Synchorn algorithm was used as the matching solver. The number of dimensions of the feature obtained by feature extraction was set to 256.

[0154] (Example) The feature extraction calculation in the first comparative example is extended to calculate the probability distribution using the method of Figure 4, and the matching solver is replaced with WeaveNet (corresponding to the matching model) proposed in Non-Patent Document 2. The first example was constructed by adding the above algorithm. Similarly, the second example was constructed by extending the feature extraction calculation in the second comparative example to calculate the probability distribution using the method of Figure 4, and then replacing the matching solver with WeaveNet proposed in Non-Patent Document 2. The third example was constructed by extending the feature extraction calculation in the third comparative example to calculate the probability distribution using the method of Figure 4, and then replacing the matching solver with WeaveNet proposed in Non-Patent Document 2. The number of layers (L) of WeaveNet in each example was set to 10. The combination selection threshold (r) was set to 0.5. The number of dimensions (Cs) of the part after feature summarization (first part) was set to 16.

[0155] (Other implementations) SGD (Stochastic Gradient Descent) optimizer uses training point cloud data Machine learning for each comparative example and each example was performed using the following: The learning rate was set to 0.015, the batch size was set to 8, and the number of epochs was set to 15. Other settings followed those described in each non-patent document (1, 5, 6). In each example, machine learning for the feature extractor and matching model was performed in an integrated manner. A commercially available computer (equipped with four Tesla V100 GPUs) was used for the machine learning for each comparative example and each example. Verification point cloud data was used to adjust the hyperparameter values ​​in each comparative example and each example. After machine learning was completed, a matching task was performed on test point cloud data using the method of each comparative example and each example. The obtained matching results were evaluated according to the following evaluation indicators.

[0156] (Evaluation indicators) The inlier ratio (IR) and non-rigid feature matching recall (NFMR) defined in Non-Patent Document 1 were used as evaluation indices for the first and second datasets ((5) and (6) of Non-Patent Document 1). The inline ratio (IR), feature matching recall (FMR), and rigid registration recall (RR) were adopted as evaluation metrics for the first and second datasets. FMR and RR are evaluation metrics proposed in Non-Patent Document 7 ((7) and (8) of Non-Patent Document 7).

[0157] (result) FIG. 17 shows the evaluation results of the methods of each comparative example and each example for each dataset in the first experimental example. As shown in FIG. 17, the examples outperformed the comparative examples under all conditions. These results show that by extending the inference of the distance between elements to non-deterministic inference, as in this embodiment, it is possible to expect an improvement in matching accuracy. In particular, it was found that by performing machine learning on the feature extractor and the matching model in an integrated manner, it is possible to expect an improvement in matching accuracy.

[0158] [Second Experimental Example] Next, Example 3-A was constructed by omitting the selection of combinations and the summarization of features in Example 3. Example 3-B was constructed by omitting only the selection of combinations in Example 3. Example 3-C was constructed by omitting only the summarization of features in Example 3. Then, in order to evaluate the calculation cost, the memory consumption during actual training and matching on the first dataset (4DMatch) was measured for Example 3, Example 3, Example 3-A, Example 3-B, and Example 3-C. Example 3, Example 3-A, Example 3-B, and Example 3-C were evaluated using the same indices (IR, NFMR) as in Example 1. Note that in Examples 3-A and 3-B, due to insufficient memory capacity, the batch size for training was changed to 1, and the computer was changed to one equipped with eight Tesla V100 GPUs. Other conditions were set to the same as in Example 1 above.

[0159] (result) Figure 18 shows the measurement results of memory consumption for the comparative example and each example in the second experimental example. As shown in Figure 18, the memory consumption of the third example was approximately twice that of the third comparative example. These results indicate that extending the inference of distances between elements to non-deterministic inference increases the processing load. On the other hand, a comparison of the third example, the third example, the third-A example, the third-B example, and the third-C example indicates that combination selection and feature summarization can reduce memory consumption while maintaining higher performance for each index than the third comparative example. A comparison of the third example and the third example and the third-A example indicates that combination selection and feature summarization can reduce memory consumption to approximately 1 / 30. These results indicate that combination selection and feature summarization can effectively reduce processing load.

[0160] [Third Experimental Example] Next, Gaussian noise (N(0,σ)) was added to the point cloud data in the second dataset (4DLoMatch) and the fourth dataset (3DLoMatch) to generate noisy point cloud data. The parameter σ was controlled from 0.0 to 1.0 in steps of 0.2. The original point cloud data and the noisy point cloud data were given to each comparative example and each example to perform a matching task. The matching results for the second dataset (4DLoMatch) were compared using non-rigid feature matching. The matching results for the fourth dataset (3DLoMatch) were particularly good. The other conditions were set to be the same as in the first example. It was determined.

[0161] (result) 19 and 20 show the second data set (4DLoMatch) and the fourth data set in the third experimental example. The evaluation results of the methods of each comparative example and each example for the dataset (3DLoMatch) are shown in Figure 1. 9 and 20, as noise increased (σ increased), the performance of each comparative example and each example decreased. However, compared to each comparative example, each example was able to suppress the performance degradation. These results show that this embodiment has high noise resistance, and therefore, it can be expected that the accuracy of matching will be improved even in a noisy environment.

[0162] [Fourth Experimental Example] Next, the values ​​of the hyperparameters in Example 3 were varied to verify the effects of the hyperparameters in Example 3. Specifically, Example 3-1-1 was constructed by changing the threshold (r) for selecting combinations in Example 3 to 0.1. Example 3-1-2 was constructed by changing the threshold (r) for selecting combinations in Example 3 to 1.0. Example 3-2-1 was constructed by changing the number of layers (L) of WeaveNet in Example 3 to 6. Example 3-2-2 was constructed by changing the number of layers (L) of WeaveNet in Example 3 to 8. Example 3-3-1 was constructed by changing the number of dimensions (Cs) of the summarized features in Example 3 to 4. Example 3-3-2 was constructed by changing the number of dimensions (Cs) of the summarized features in Example 3 to 64. Using the second dataset (4DLoMatch) and the fourth dataset (3DLoMatch), the methods of each experimental example were evaluated using the same indicators as in Example 1.

[0163] (result) FIG. 21 shows the second dataset (4DLoMatch) and the fourth dataset in the fourth experimental example. The evaluation results of the methods of the third comparative example and each example against (3DLoMatch) are shown in FIG. As shown in the figure, it was found that performance can be improved by increasing the number of layers in the matching model (WeaveNet). In addition, it was found that the selection threshold (r) and the number of dimensions of the summarized features (Cs) can lead to a decrease in performance if they are too large or too small. It was found that it is preferable to set the selection threshold (r) and the number of dimensions of the summarized features (Cs) appropriately.

[0164] [5th ​​Experimental Example] First, as a preliminary step, we prepared the fifth dataset (Surreal) and the sixth dataset (Surreal (train) / SHREC (test)).

[0165] The fifth dataset (Surreal) was composed of the Surreal dataset (point cloud data of human body shapes) proposed in Non-Patent Document 8 (rigid body setting). 115K training sample pairs were generated by randomly combining 230K training samples. 100 test sample pairs were also generated by randomly rotating and translating 100 test samples.

[0166] The sixth dataset (Surreal(train) / SHREC(test)) is proposed in Non-Patent Document 9. The SHREC dataset (non-rigid setting) was added to the fifth dataset. For the sixth dataset, the Surreal dataset was used for training and the SHREC dataset was used for testing. 860 samples in the SHREC dataset were randomly combined. By combining these data, 430 test sample pairs were generated. The point cloud data contained 1024 points.

[0167] (Comparative Example) In Comparative Example 4-1, we used the model (CorrNet3D) proposed in Non-Patent Document 10 and the matching module. By combining these modules, we configured the calculations from feature extraction to matching for each input. In the comparative example 4-1, we used a model (CorrNet3D) with an unsupervised training option. The module used was the DeSmooth module proposed in Non-Patent Document 10. Comparative Example 4-2 uses a model (CorrNet3D) with a supervised training option. The other configurations of the 4-2 comparative example were set to be the same as those of the 4-1 comparative example.

[0168] (Example) Example 4-1 was constructed by extending the feature extraction calculation in Comparative Example 4-1 to calculate a probability distribution using the method of Figure 4, and then replacing the matching module with WeaveNet proposed in Non-Patent Document 2. Similarly, Example 4-2 was constructed by extending the feature extraction calculation in Comparative Example 4-2 to calculate a probability distribution using the method of Figure 4, and then replacing the matching module with WeaveNet proposed in Non-Patent Document 2. As with Example 1, the number of layers (L) of WeaveNet in each example was set to 10. The threshold (r) for selecting combinations was set to 0.5. The number of dimensions of the summarized features (Cs) was set to 16.

[0169] (Other implementations) Machine learning for each comparative example and each example was performed using the Adam optimizer and training point cloud data. The learning rate was set to 1e-4, the batch size to 10, and the number of epochs to 100. Other settings followed those described in Non-Patent Document 10. In each example, machine learning for the feature extractor and matching model was performed in an integrated manner. A commercially available computer was used for the machine learning for each comparative example and example. After the machine learning was completed, a matching task was performed on test point cloud data using the method of each comparative example and example. The obtained matching results were evaluated according to the following evaluation indicators.

[0170] (Evaluation indicators) As an evaluation index for each dataset, we adopted Corr (corresponding percentage) defined in Non-Patent Document 10. Corr (corresponding percentage) was calculated under controlled tolerances.

[0171] (result) Figures 22 and 23 show the evaluation results of the methods of each comparative example and each example for the fifth and sixth datasets in the fifth experimental example (Figure 22: fifth dataset, Figure 23: sixth dataset). Figures 24 and 25 show combinations of points that failed to be matched (Figure 24) and combinations of points that were successful (Figure 25) for the test sample using the method of the 4-1 comparative example. Figures 26 and 27 show combinations of points that failed to be matched (Figure 26) and combinations of points that were successful (Figure 27) for the test sample using the method of the 4-1 example.

[0172] As shown in the figures, the Example outperformed the Comparative Example under all conditions. In particular, the Example significantly outperformed the Comparative Example in the low tolerance range. These results demonstrate that the present embodiment can be expected to improve matching accuracy. Furthermore, the significant improvement in performance in the low tolerance range demonstrates that the present embodiment can effectively discover pairs based on similarities between samples.

[0173] This specification includes the following disclosure. [Appendix 1] obtaining a probability distribution (40) of distances between a first sample (30) of a first input comprising a plurality of elements and a second sample (35) of a second input comprising a plurality of elements; providing the acquired probability distribution (40) to a matching model (50) and executing a calculation process of the matching model (50) to acquire a matching result between the elements of the first sample (30) and the second sample (35); and outputting information about the obtained matching result; A control unit (11) configured to execute Collation device (1). [Appendix 2] The matching model (50) is composed of differentiable operations, Obtaining the probability distribution (40) of the distances includes: Calculating feature quantities (31, 36) of each of the elements contained in the first sample (30) and the second sample (35), respectively, using a feature extractor (60) trained by machine learning; and calculating a probability distribution (40) of the distance between the first sample (30) and the second sample (35) from the calculated feature amounts (31, 36) of the elements included in the first sample (30) and the second sample (35), respectively; It consists of 1. A verification device (1) as described in appendix 1. [Appendix 3] The matching model (50) is a machine learning model trained by machine learning integrated with the feature extractor (60). 2. A verification device (1) according to claim 2. [Appendix 4] Calculating the probability distribution (40) of the distance from the feature amounts (31, 36) of each element calculating an evaluation value between each of the elements of the first sample (30) and the second sample (35) from the calculated feature amounts (31, 36); selecting one or more combinations from the combinations of the elements of the first sample (30) and the second sample (35) based on the calculated evaluation value; and calculating a probability distribution (40) of the distance between each element of the one or more selected combinations from the calculated feature amounts (31, 36); It is composed of the probability distribution (40) of the distance between the first sample (30) and the second sample (35) is constituted by a probability distribution (40) of the distance between each element of the one or more selected combinations; A verification device (1) according to appendix 2 or appendix 3. [Appendix 5] the first input is first point cloud data (D1); the second input is second point cloud data (D2), The matching between the elements is a matching between points included in the first point cloud data (D1) and points included in the second point cloud data (D2). A verification device (1) according to any one of appendices 1 to 4. [Appendix 6] the first input is a first image (D3); the second input is a second image (D4); The matching between the elements is a matching between points included in the first image (D3) and points included in the second image (D4). A verification device (1) according to any one of appendices 1 to 4. [Appendix 7] The first input is a first set of people (D5), The second input is a second set of people (D6), The matching between the elements is a matching between a person included in the first person set (D5) and a person included in the second person set (D6). A verification device (1) according to any one of appendices 1 to 4. [Appendix 8] The first input is a set of people (D7), The second input is an entity set (D8), The matching between the elements is a matching between the people included in the person set (D7) and the objects included in the object set (D8). A verification device (1) according to any one of appendices 1 to 4. [Appendix 9] The first input is a first object set (D9), The second input is a second set of entities (D10), The matching between the elements is a matching between the entities included in the first entity set (D9) and the entities included in the second entity set (D10). A verification device (1) according to any one of appendices 1 to 4. [Appendix 10] The computer (1) obtaining a probability distribution (40) of distances between a first sample (30) of a first input comprising a plurality of elements and a second sample (35) of a second input comprising a plurality of elements; a step of providing the acquired probability distribution (40) to a matching model (50) and executing a calculation process of the matching model (50) to acquire a matching result between the elements of the first sample (30) and the second sample (35); outputting information about the obtained matching result; To execute Matching method. [Appendix 11] On the computer (1), obtaining a probability distribution (40) of distances between a first sample (30) of a first input comprising a plurality of elements and a second sample (35) of a second input comprising a plurality of elements; The acquired probability distribution (40) is given to a matching model (50), and the matching model (50) ) to obtain a matching result between the elements of the first sample (30) and the second sample (35); outputting information about the obtained matching result; In order to execute Matching program (81). [Appendix 12] Computer (2) controlling the machine learning of a feature extractor (60) in conjunction with a matching model (50); outputting the results of the machine learning; Run The feature extractor (60) is configured to calculate a feature value for each element included in the input; the matching model (50) is configured to derive a matching result between elements included in the first input and the second input from a probability distribution (45) of distances between the first input and the second input by a differentiable operation; The machine learning calculating, using the feature extractor (60), feature quantities (34, 39) of each element included in a first training sample (33) of the first input and a second training sample (38) of the second input, respectively; calculating a probability distribution (45) of the distance between the first training sample (33) and the second training sample (38) from the calculated feature amounts (34, 39) of each of the elements included in the first training sample (33) and the second training sample (38), respectively; providing the calculated probability distribution (45) of the distance to the matching model (50) and executing a calculation process of the matching model (50) to derive a matching result between the elements of the first training sample (33) and the second training sample (38); and training the feature extractor (60) so that the derived matching results approach true values ​​(70) corresponding to the first training sample (33) and the second training sample (38); Including, Machine learning methods. [Appendix 13] training the feature extractor (60) comprises jointly training the feature extractor (60) and the matching model (50); 13. The machine learning method of claim 12. [Appendix 14] Calculating a probability distribution (45) of the distance from the feature amount (34, 39) of each element calculating an evaluation value between each of the elements of the first training sample (33) and the second training sample (38) from the calculated feature amounts (34, 39); selecting one or more combinations from the combinations of the elements of the first training sample (33) and the second training sample (38) based on the calculated evaluation value; and calculating a probability distribution (45) of the distance between each element of the one or more selected combinations from the calculated feature amounts (34, 39); It is composed of the probability distribution (45) of the distance between the first training sample (33) and the second training sample (38) is constituted by a probability distribution (45) of the distance between each element of the one or more selected combinations; 14. The machine learning method according to claim 12 or 13. [Appendix 15] the first input is first point cloud data (D1); the second input is second point cloud data (D2), The matching between the elements is a matching between points included in the first point cloud data (D1) and points included in the second point cloud data (D2). 15. The machine learning method of any one of appendices 12 to 14. [Appendix 16] the first input is a first image (D3); the second input is a second image (D4); The matching between the elements is a matching between points included in the first image (D3) and points included in the second image (D4). 15. The machine learning method of any one of appendices 12 to 14. [Appendix 17] The first input is a first set of people (D5), The second input is a second set of people (D6), The matching between the elements is a matching between a person included in the first person set (D5) and a person included in the second person set (D6). 15. The machine learning method of any one of appendices 12 to 14. [Appendix 18] The first input is a set of people (D7), The second input is an entity set (D8), The matching between the elements is a matching between the people included in the person set (D7) and the objects included in the object set (D8). 15. The machine learning method of any one of appendices 12 to 14. [Appendix 19] The first input is a first object set (D9), The second input is a second set of entities (D10), The matching between the elements is a matching between the entities included in the first entity set (D9) and the entities included in the second entity set (D10). 15. The machine learning method of any one of appendices 12 to 14. [Appendix 20] Controlling the machine learning of a feature extractor (60) in conjunction with a matching model (50); and outputting the results of the machine learning; a control unit (21) configured to execute the The feature extractor (60) is configured to calculate a feature value for each element included in the input; the matching model (50) is configured to derive a matching result between elements included in the first input and the second input from a probability distribution (45) of distances between the first input and the second input by a differentiable operation; The machine learning calculating, using the feature extractor (60), feature quantities (34, 39) of each element included in a first training sample (33) of the first input and a second training sample (38) of the second input, respectively; calculating a probability distribution (45) of the distance between the first training sample (33) and the second training sample (38) from the calculated feature amounts (34, 39) of each of the elements included in the first training sample (33) and the second training sample (38), respectively; providing the calculated probability distribution (45) of the distance to the matching model (50) and executing a calculation process of the matching model (50) to derive a matching result between the elements of the first training sample (33) and the second training sample (38); and training the feature extractor (60) so that the derived matching results approach true values ​​(70) corresponding to the first training sample (33) and the second training sample (38); Including, Machine learning device (2). [Explanation of symbols]

[0174] 1...collation device, 11...control unit, 12...storage unit, 13...communication interface, 14...input device, 15...output device, 16...drive, 81...collation program, 91...storage medium, 111...acquisition unit, 112...collation unit, 113...output processing unit, 30...first sample, 35...second sample, 31·36…Features, 40...Probability distribution, 50...Matching model, 60...Feature extractor, 2...machine learning device, 21...control unit, 22...storage unit, 23...communication interface, 24...input device, 25...output device, 26...drive, 82...machine learning program, 92...storage medium, 211...acquisition unit, 212...learning processing unit, 213...output processing unit, 33...first training sample, 38...second training sample, 34·39…Features, 45…Probability distribution, 70…True value

Claims

1. obtaining a probability distribution of distances between a first sample of a first input comprising a plurality of elements and a second sample of a second input comprising a plurality of elements; providing the acquired probability distribution to a matching model and performing a calculation process on the matching model to acquire a matching result between the elements of the first sample and the second sample; and outputting information about the obtained matching result; a control unit configured to perform Collation device.

2. the matching model is composed of differentiable operations; Obtaining the probability distribution of distances includes: calculating a feature amount of each of the elements included in each of the first sample and the second sample using a feature extractor trained by machine learning; and calculating a probability distribution of the distance between the first sample and the second sample from the calculated feature amounts of the elements included in the first sample and the second sample; It consists of The verification device according to claim 1 .

3. The matching model is a machine learning model trained by machine learning integrated with the feature extractor. The verification device according to claim 2 .

4. Calculating the probability distribution of the distances from the feature amounts of the elements calculating an evaluation value between the elements of the first sample and the second sample from the calculated feature amount; selecting one or more combinations from the combinations of the elements of the first sample and the second sample based on the calculated evaluation value; and calculating a probability distribution of the distance between elements of each of the one or more selected combinations from the calculated feature amounts; It is composed of the probability distribution of the distance between the first sample and the second sample is configured by a probability distribution of the distance between each element of the one or more selected combinations. The verification device according to claim 2 .

5. the first input is first point cloud data; the second input is second point cloud data; the matching between elements is a matching between points included in the first point cloud data and points included in the second point cloud data; The collation device according to any one of claims 1 to 4.

6. the first input is a first image; the second input is a second image; the matching between elements is a matching between points included in the first image and points included in the second image; The collation device according to any one of claims 1 to 4.

7. the first input is a first set of people; the second input is a second set of people; The matching between elements is a matching between a person included in the first set of people and a person included in the second set of people. The collation device according to any one of claims 1 to 4.

8. the first input is a set of people; the second input is a set of entities; The matching between the elements is a matching between people included in the set of people and objects included in the set of objects. The collation device according to any one of claims 1 to 4.

9. the first input is a first set of entities; the second input is a second set of entities; The matching between elements is a matching between an entity included in the first entity set and an entity included in the second entity set. The collation device according to any one of claims 1 to 4.

10. The computer obtaining a probability distribution of distances between a first sample of a first input comprising a plurality of elements and a second sample of a second input comprising a plurality of elements; providing the acquired probability distribution to a matching model and executing a calculation process of the matching model to acquire a matching result between the elements of the first sample and the second sample; outputting information about the obtained matching result; To execute Matching method.

11. On the computer, obtaining a probability distribution of distances between a first sample of a first input comprising a plurality of elements and a second sample of a second input comprising a plurality of elements; providing the acquired probability distribution to a matching model and executing a calculation process of the matching model to acquire a matching result between the elements of the first sample and the second sample; outputting information about the obtained matching result; In order to execute Matching program.

12. The computer controlling machine learning of a feature extractor in conjunction with a matching model; outputting the results of the machine learning; Run the feature extractor is configured to calculate a feature value for each element included in the input; the matching model is configured to derive a matching result between elements included in the first input and the second input from a probability distribution of a distance between the first input and the second input by a differentiable operation; The machine learning calculating, using the feature extractor, feature amounts of each element included in a first training sample of the first input and a second training sample of the second input; From the calculated feature amounts of the elements included in the first training sample and the second training sample, calculating a probability distribution of the distances; providing the calculated probability distribution of the distances to the matching model and performing a calculation process on the matching model to derive a matching result between the elements of the first training sample and the second training sample; and training the feature extractor so that the derived matching results approach true values ​​corresponding to the first training sample and the second training sample; Including, Machine learning methods.

13. training the feature extractor comprises jointly training the feature extractor and the matching model; The machine learning method of claim 12.

14. Calculating a probability distribution of the distances from the feature amounts of the elements calculating an evaluation value between each of the elements of the first training sample and the second training sample from the calculated feature amount; selecting one or more combinations from the combinations of the elements of the first training sample and the second training sample based on the calculated evaluation value; and calculating a probability distribution of the distance between elements of each of the one or more selected combinations from the calculated feature amounts; It is composed of the probability distribution of the distance between the first training sample and the second training sample is composed of a probability distribution of the distance between each element of the one or more selected combinations. The machine learning method of claim 12.

15. the first input is first point cloud data; the second input is second point cloud data; the matching between elements is a matching between points included in the first point cloud data and points included in the second point cloud data; The machine learning method according to any one of claims 12 to 14.

16. the first input is a first image; the second input is a second image; the matching between elements is a matching between points included in the first image and points included in the second image; The machine learning method according to any one of claims 12 to 14.

17. the first input is a first set of people; the second input is a second set of people; The matching between elements is a matching between a person included in the first set of people and a person included in the second set of people. The machine learning method according to any one of claims 12 to 14.

18. the first input is a set of people; the second input is a set of entities; The matching between the elements is a matching between people included in the set of people and objects included in the set of objects. The machine learning method according to any one of claims 12 to 14.

19. the first input is a first set of entities; the second input is a second set of entities; The matching between elements is a matching between an entity included in the first entity set and an entity included in the second entity set. The machine learning method according to any one of claims 12 to 14.

20. Controlling the machine learning of the feature extractor in conjunction with the matching model; and outputting the results of the machine learning; a control unit configured to perform the feature extractor is configured to calculate a feature value for each element included in the input; the matching model is configured to derive a matching result between elements included in the first input and the second input from a probability distribution of a distance between the first input and the second input by a differentiable operation; The machine learning calculating, using the feature extractor, feature amounts of each element included in a first training sample of the first input and a second training sample of the second input; calculating a probability distribution of distances between the first training sample and the second training sample from the calculated feature amounts of the elements included in the first training sample and the second training sample; providing the calculated probability distribution of the distances to the matching model and performing a calculation process on the matching model to derive a matching result between the elements of the first training sample and the second training sample; and training the feature extractor so that the derived matching results approach true values ​​corresponding to the first training sample and the second training sample; Including, Machine learning device.

Images