Model generation method, model generation device, inference program, and inference device

Abstract

To improve efficiency of inference processing for data having a graph structure.SOLUTION: A model generation device executes machine learning of an inference model comprising a preprocessing module and a graph inference module. The preprocessing module comprises a feature extractor and a selection module. The feature extractor calculates a feature amount of each element belonging to each of a plurality of sets included in an input graph. The selection module selects one or more branches extending from each element on the basis of the calculated feature amount of each element, and generates, for each set, graph information indicating the calculated feature amount of each element and the branch selection result. The graph inference module is differentiable, and infers a task solution with respect to an input graph from the graph information generated for each set.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a model generation method, a model generation device, an inference program, and an inference device. [Background technology]

[0002] When solving various inference tasks such as matching and pathfinding, graphs are often used to represent various types of information. Graphs are basically composed of elements (vertices / nodes) and edges. Edges indicate relationships between elements. Depending on the inference task, graphs with various structures, such as directed graphs, bipartite graphs, vertex feature graphs, and hypergraphs, can be used.

[0003] As an example, a bipartite graph can be used to solve a stable matching problem. A stable matching problem is, for example, a two-sided market allocation problem, such as matching between job seekers and employers. A bipartite graph is a graph constructed by dividing a set of elements (vertices) into two subsets, with no edges between elements in each subset. Each subset may correspond to a party to which a matching target belongs, and elements (vertices) in each subset may correspond to matching targets (e.g., job seekers / employers). Each target may have a degree of desirability (e.g., preference ranking) for targets belonging to the other party, and the desirability may be represented by an edge.

[0004] A matching can be represented by a branch set consisting of branches that represent combinations of elements belonging to one subset and elements belonging to the other subset, and that are assigned so that elements are not shared. If a branch set that constitutes a matching contains a combination of elements that is more desirable than a combination of branches belonging to the branch set, the branch representing this combination blocks the matching (i.e., indicates the existence of other matchings that are more desirable than the matching represented by the branch set). In one example of a stable matching problem, a combination of elements is found that does not have such blocking pairs and satisfies an arbitrary metric (e.g., fairness).

[0005] As another example, when inferring correspondences between multiple pieces of point cloud data (such as 2D / 3D models) (e.g., identifying identical elements), a vertex feature graph can be used. A vertex feature graph is a graph configured so that each element (vertex) has an attribute. Each point constituting the point cloud data corresponds to each element. Inferring correspondences can correspond to, for example, point cloud registration, optical flow estimation, stereo matching, etc.

[0006] As a typical example, consider a situation where point cloud data represents a two-dimensional or three-dimensional model, and the same points between two pieces of point cloud data are identified (i.e., a combination of points having a corresponding relationship is identified). In this case, the candidates for points having a corresponding relationship with each point included in one piece of point cloud data are basically all points in the other piece of point cloud data. An example of an inference task consists of identifying points having a corresponding relationship from among the candidates (i.e., estimating a combination of points having a corresponding relationship from multiple candidates).

[0007] In addition, when the relationships between elements are unknown, such as when inferring the correspondence between point cloud data, the graph provided before inference (in the above example, multiple point cloud data) does not need to have edges. In this case, when solving the inference task, it can be assumed that edges are provided for all candidate combinations of elements that may have correspondence.

[0008] Various algorithms have been developed to solve tasks represented by such graphs. The GS algorithm proposed in Non-Patent Document 1 is known as an example of a classic method for solving the stable matching problem. However, methods using such specific algorithms lack versatility because they become unusable if the content of the task changes even slightly. In addition, many parts are manually configured, which increases the cost of generating a solver.

[0009] Therefore, in recent years, research has been conducted into methods using machine learning models such as neural networks as a general-purpose method for solving tasks represented by graphs. For example, Non-Patent Document 2 proposes solving stable matching problems using a five-layer neural network. Non-Patent Document 3 proposes a method for solving tasks represented by graphs using WeaveNet, which is composed of machine learning models. According to such a method using a machine learning model, by preparing training data according to the task, it is possible to generate a trained model (solver) that has acquired the ability to solve the task. This makes it possible to increase the versatility of the generated model and reduce the cost of generating the solver. [Prior art documents] [Non-patent literature]

[0010] [Non-Patent Document 1] David Gale, Lloyd S Shapley, “College admissions and the stability of marriage”, The American Mathematical Monthly, 69(1):9-15, 1962. [Non-patent document 2] Shira Li, “Deep Learning for Two-Sided Matching Markets”, PhD thesis, Harvard University, 2019. [Non-licensed document 3] shutdown <url: https: openreview.net forum?id="ktHKpsbsxx"> Summary of the Invention [Problem to be solved by the invention]

[0011] The present inventors have found that the methods proposed in Non-Patent Documents 2 and 3, etc., for solving inference tasks for graph-structured data using machine learning models have the following problem. That is, the more elements there are, the more edges there are to be referenced in the inference process. As an example, in the above stable matching problem, if one subset has N elements and the other subset has M elements, and desirability is given between all elements by directed edges, the number of edges to be referenced will be 2MN. As another example, in the case of solving the task of identifying identical points between the above two sets of point cloud data, if the number of points (elements) included in each set of point cloud data is N, and if a combination of identical points is to be identified from all candidates (brute force), the number of edges (candidates) to be referenced will be N. 2 This increases the computational load for solving the inference task, which may result in a decrease in processing efficiency.

[0012] In one aspect, the present invention has been made in consideration of the above circumstances, and its purpose is to provide a technology that improves processing efficiency when solving inference tasks for data having a graph structure using a machine learning model. [Means for solving the problem]

[0013] In order to solve the above-mentioned problems, the present invention employs the following configuration.

[0014] That is, a model generation method according to one aspect of the present invention is an information processing method in which a computer executes the steps of acquiring a plurality of training graphs and performing machine learning of an inference model using the acquired plurality of training graphs. The machine learning system includes a preprocessing module and a graph inference module. The preprocessing module includes a feature extractor and a selection module. The feature extractor is configured to calculate features of each element belonging to each of a plurality of sets included in an input graph. The selection module is configured to select one or more edges extending from each element based on the calculated features of each element, and to generate, for each set, graph information indicating the calculated features of each element and the selected one or more edges. The graph inference module is configured to be differentiable and to infer a solution to a task for the input graph from the generated graph information of each set. The machine learning is configured by inputting each training graph as the input graph to the preprocessing module and training the inference model so that the inference results obtained from the graph inference module match the correct answer to the task for each training graph.

[0015] The model generation method according to this configuration can generate a trained inference model equipped with a preprocessing module. The preprocessing module calculates the feature values ​​of each element of the input graph and, based on the calculated feature values ​​of each element, can select edges (combinations of elements) to be referenced when solving an inference task. In other words, even if the number of elements included in the data increases, the number of edges to be referenced in the inference process can be reduced by selecting edges using the preprocessing module. Therefore, the trained inference model can improve the efficiency of processing when solving an inference task for data with a graph structure.

[0016] In the model generation method according to the above aspect, selecting the one or more branches based on the feature values ​​of the elements may be configured by calculating likelihoods between each element and each candidate element that is a candidate for connecting a branch from the element, and selecting the one or more branches according to the calculated likelihoods. With this configuration, it is possible to appropriately select branches to be referenced when solving an inference task based on the likelihoods, which is expected to improve the efficiency and accuracy of the inference process.

[0017] In the model generation method according to the above aspect, each of the training graphs may be a directed bipartite graph, and a plurality of elements included in the directed bipartite graph may be divided to belong to one of two subsets. The plurality of sets of the input graph may be composed of the two subsets of the directed bipartite graph. Calculating the feature of each of the elements may be performed by calculating the feature of each of the elements from the features of an edge extending from each of the elements in the directed bipartite graph as a starting point. This configuration makes it possible to improve the efficiency of the process of solving an inference task (inferring a solution) when the inference task is represented by a directed bipartite graph.

[0018] In the model generation method according to the above aspect, the task may be a matching task between two parties, determining an optimal pairing of objects belonging to each party. The two subsets of the directed bipartite graph may correspond to the two parties in the matching task. The elements belonging to each subset of the directed bipartite graph may correspond to the objects belonging to each party. This configuration can improve the efficiency of the process of inferring a solution for a two-sided matching represented by a directed bipartite graph.

[0019] In the model generation method according to the above aspect, each of the training graphs may be an undirected bipartite graph, and a plurality of elements included in the undirected bipartite graph may be divided to belong to one of two subsets. The plurality of sets of the input graph may be composed of the two subsets of the undirected bipartite graph. Calculating the feature of each element may be performed by calculating the feature of each element from the feature of an edge connecting to each element in the undirected bipartite graph. This configuration can improve the efficiency of processing to solve an inference task when the inference task is represented by an undirected bipartite graph.

[0020] In the model generation method according to the above aspect, the task may be a matching task between two parties, determining an optimal pairing of objects belonging to each party. The two subsets of the undirected bipartite graph may correspond to the two parties in the matching task. The elements belonging to each subset of the undirected bipartite graph may correspond to the objects belonging to each party. This configuration can improve the efficiency of processing to infer a solution for one-sided matching represented by an undirected bipartite graph.

[0021] In the model generation method according to the above aspect, each of the training graphs may be a directed graph. The multiple sets of the input graph may be composed of a first set and a second set. The elements belonging to the first set may correspond to starting points of directed edges constituting the directed graph, and the elements belonging to the second set may correspond to ending points of the directed edges. Calculating the feature values ​​of each element may be performed by calculating the feature values ​​of the elements belonging to the first set from features of directed edges emanating from the elements belonging to the first set, and calculating the feature values ​​of the elements belonging to the second set from features of directed edges emanating from the elements belonging to the second set. This configuration can improve the efficiency of processing to solve an inference task when the inference task is represented by a directed graph.

[0022] In the model generation method according to the above aspect, each of the training graphs may be an undirected graph. The multiple sets of the input graph may be composed of a first set and a second set. Each element belonging to each of the first set and the second set may correspond to each element constituting the undirected graph. Calculating the feature of each element may be performed by calculating the feature of each element from the feature of an edge connecting to each element. This configuration can improve the efficiency of processing to solve an inference task when the inference task is represented by an undirected graph.

[0023] In the model generation method according to the above aspect, each training graph may be configured such that each element included in the training graph has an attribute. Calculating the feature value of each element may be configured by calculating the feature value of each element from the attribute of each element. This configuration can improve the efficiency of a process for solving an inference task when the inference task is represented by a vertex feature graph in which vertices (elements) have features (attributes). The task may be estimating a relationship between elements belonging to each set. Estimating the relationship between elements may be, for example, point cloud alignment, optical flow estimation, stereo matching, or the like. This improves the efficiency of a process for estimating the relationship between elements.

[0024] In the model generation method according to the above aspect, each of the training graphs may be configured so that each element included in the training graph has an attribute. A relationship between the elements included in each of the training graphs may be defined. Calculating the feature value of each element may be configured by calculating the feature value of each element from information indicating the attributes and relationships possessed by each of the elements. This configuration can improve the efficiency of processing to solve an inference task in a situation where the inference task is represented by a vertex feature graph in which relationships between vertices are given in advance.

[0025] In the model generation method according to the above aspect, each of the training graphs may be a hypergraph, and a plurality of elements included in the hypergraph may be divided to belong to one of three or more subsets. The plurality of sets of the input graph may be configured by the three or more subsets of the hypergraph. Calculating the feature of each element may include calculating the feature of each element from the feature of an edge connected to each element of the hypergraph. According to this configuration, in a situation where an inference task is represented by a hypergraph, it is possible to improve the efficiency of the process of solving the inference task.

[0026] Furthermore, the present invention is not limited to the above-described model generation method. One aspect of the present invention may be an inference method including the step of inferring a task solution for a graph using a trained inference model generated by the model generation method according to any of the above-described aspects. For example, an inference method according to one aspect of the present invention is an information processing method in which a computer executes the steps of acquiring a target graph, inferring a task solution for the acquired target graph using an inference model trained by machine learning, and outputting information related to the result of inferring the task solution. The inference model has a similar configuration to that described above. Inferring a task solution for the target graph is performed by inputting the target graph as the input graph to the preprocessing module and obtaining the result of inferring the task solution from the graph inference module. This configuration can improve the efficiency of processing when inferring a task solution using a trained inference model. Note that the inference method may be interpreted as, for example, a matching method, a prediction method, or the like, depending on the type of task in an application scenario. The same applies to other aspects (e.g., an inference device, etc.).

[0027] Furthermore, as another aspect of each of the model generation method and inference method according to the above embodiments, one aspect of the present invention may be an information processing device (model generation device, inference device) that realizes all or part of the above configurations, or may be a program, or may be a storage medium that stores such a program and is readable by a computer or other device, machine, etc. Here, a storage medium that is readable by a computer, etc. is a medium that stores information such as a program by electrical, magnetic, optical, mechanical, or chemical action. Another aspect of the present invention may be an inference system configured by the model generation device and the inference device.

[0028] For example, a model generation device according to one aspect of the present invention includes an acquisition unit configured to acquire a plurality of training graphs, and a learning processing unit configured to perform machine learning of an inference model using the acquired plurality of training graphs. Also, for example, a model generation program according to one aspect of the present invention is a program for causing a computer to execute the steps of acquiring a plurality of training graphs and performing machine learning of an inference model using the acquired plurality of training graphs. The inference model is configured in the same manner as described above. The machine learning is performed by inputting each of the training graphs as the input graph to the preprocessing module, and training the inference model so that the inference result obtained from the graph inference module matches the correct answer to the task for each of the training graphs.

[0029] For example, an inference device according to one aspect of the present invention includes an acquisition unit configured to acquire a target graph, an inference unit configured to infer a task solution for the acquired target graph using an inference model trained by machine learning, and an output unit configured to output information related to the result of inferring the task solution. Also, for example, an inference program according to one aspect of the present invention is a program for causing a computer to execute the steps of acquiring a target graph, inferring a task solution for the acquired target graph using an inference model trained by machine learning, and outputting information related to the result of inferring the task solution. The inference model is configured in the same manner as above. Inferring the task solution for the target graph includes inputting the target graph as the input graph to the preprocessing module and obtaining the result of inferring the task solution from the graph inference module. [Effects of the Invention]

[0030] According to the present invention, it is possible to improve the efficiency of processing when solving an inference task for data having a graph structure using a machine learning model. [Brief explanation of the drawings]

[0031] [Figure 1] FIG. 1 shows a schematic diagram of an example of a situation in which the present invention is applied. [Figure 2] FIG. 2 is a schematic diagram showing an example of the configuration of an inference model according to an embodiment. [Figure 3] FIG. 3 is a diagram illustrating an example of the calculation process of the pre-processing module according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of a hardware configuration of a model generating device according to an embodiment. [Figure 5] FIG. 5 is a diagram illustrating an example of a hardware configuration of an inference device according to an embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of a software configuration of the model generating device according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of the software configuration of the inference device according to the embodiment. [Figure 8] FIG. 8 is a flowchart illustrating an example of a processing procedure of the model generating device according to the embodiment. [Figure 9] FIG. 9 is a flowchart illustrating an example of a processing procedure of the inference device according to the embodiment. [Figure 10] FIG. 10 schematically illustrates an example of another situation to which the present invention is applied (a situation in which a task is represented by a directed bipartite graph). [Figure 11] FIG. 11 schematically illustrates an example of another situation to which the present invention is applied (a situation in which a task is represented by an undirected bipartite graph). [Figure 12] FIG. 12 illustrates a schematic example of another situation to which the present invention is applied (a situation in which a task is represented by a general directed graph). [Figure 13] FIG. 13 illustrates a schematic example of another situation to which the present invention is applied (a situation in which a task is represented by a general undirected graph). [Figure 14] FIG. 14 illustrates a schematic example of another situation to which the present invention is applied (a situation in which the task of inferring relationships between elements is represented by a vertex feature graph). [Figure 15] FIG. 15 illustrates a schematic example of another situation to which the present invention is applied (a situation in which a task is represented by a vertex feature graph in which relationships between elements are given). [Figure 16] FIG. 16 shows a schematic example of another situation to which the present invention is applied (a situation in which a task is represented by a hypergraph). [Figure 17] FIG. 17 shows the experimental results (inference accuracy at a set allowable error) of the example and the comparative example for the first data set. [Figure 18] FIG. 18 shows the experimental results (inference accuracy at a set allowable error) of the example and comparative example for the second data set. [Figure 19A] FIG. 19A shows the inference results of the comparative example for the first sample of the second data set when the tolerance is set to 0.00. [Figure 19B] FIG. 19B shows the inference results of the example for the first sample of the second data set when the tolerance is set to 0.00. [Figure 19C] FIG. 19C shows the inference results of the comparative example for the first sample of the second data set when the tolerance is set to 0.06. [Figure 19D] FIG. 19D shows the inference results of the example for the first sample of the second data set when the tolerance is set to 0.06. [Figure 20A] FIG. 20A shows the inference results of the comparative example for the second sample of the second data set when the tolerance is set to 0.00. [Figure 20B] FIG. 20B shows the inference results of the example for the second sample of the second data set when the tolerance is set to 0.00. [Figure 20C] FIG. 20C shows the inference results of the comparative example for the second sample of the second dataset when the tolerance is set to 0.06. [Figure 20D] FIG. 20D shows the inference results of the example for the second sample of the second data set when the tolerance is set to 0.06. DETAILED DESCRIPTION OF THE INVENTION

[0032] An embodiment according to one aspect of the present invention (hereinafter also referred to as "the present embodiment") will be described below with reference to the drawings. However, the present embodiment described below is merely an example of the present invention in all respects. Needless to say, various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiment may be appropriately adopted. Note that, although data appearing in the present embodiment are described in natural language, more specifically, they are specified using computer-recognizable pseudo-language, commands, parameters, machine language, etc.

[0033] §1 Application Examples 1 is a diagram illustrating an example of a situation in which the present invention is applied. As shown in FIG. 1, an inference system 100 according to this embodiment includes a model generating device 1 and an inference device 2.

[0034] (Model generation device) The model generation device 1 according to this embodiment is one or more computers configured to generate a trained inference model 5 by performing machine learning. Specifically, the model generation device 1 acquires multiple training graphs 30. The type of each training graph 30 may be selected appropriately depending on the task-solving ability of the inference model 5 to be acquired. Each training graph 30 may be, for example, a directed bipartite graph, an undirected bipartite graph, a general directed graph, a general undirected graph, a vertex feature graph, a hypergraph, or the like. The data format of each training graph 30 may be selected appropriately depending on the embodiment. Each training graph 30 may be composed of, for example, image data, point cloud data, or other data indicating a graph structure. The model generation device 1 performs machine learning on the inference model 5 using the acquired multiple training graphs 30. This allows the trained inference model 5 to be generated.

[0035] (Inference device) On the other hand, the inference device 2 according to this embodiment is one or more computers configured to infer a task solution for a graph using a trained inference model 5. Specifically, the inference device 2 acquires a target graph 221. The target graph 221 is a graph from which a task solution is to be inferred. The type of the target graph 221 may be appropriately selected depending on the capabilities acquired by the trained inference model 5, similar to the training graph 30. The data format of the target graph 221 may also be appropriately selected depending on the embodiment, similar to the training graph 30. The target graph 221 may be, for example, a directed bipartite graph, an undirected bipartite graph, a general directed graph, a general undirected graph, a vertex feature graph, a hypergraph, or the like. The target graph 221 may be composed of, for example, image data, point cloud data, or the like. The inference device 2 uses a trained inference model 5 through machine learning to infer a task solution for the acquired target graph 221. The inference device 2 then outputs information related to the result of inferring the task solution.

[0036] (inference model) 2 is a schematic diagram showing an example of the configuration of the inference model 5 according to this embodiment. As shown in FIGS. 1 and 2, the inference model 5 according to this embodiment is composed of a preprocessing module 50 and a graph inference module 55. The preprocessing module 50 includes a feature extractor 501 and a selection module 503.

[0037] The feature extractor 501 according to this embodiment extracts the features belonging to each of the multiple sets included in the input graph. In one example, the feature extractor 501 may be configured to calculate, for each set, the feature amount of each element from features related to each element belonging to each set in the input graph. Elements correspond to vertices (nodes) of the graph. The features related to each element may be, for example, features of the edges connecting to each element, features (attributes) of each element itself, etc.

[0038] The feature extractor 501 may be configured by a machine learning model having one or more parameters for executing a computation process to extract features, the parameters being adjusted by machine learning. The feature extractor 501 may be expressed, for example, by a functional expression. This may make the feature extractor 501 differentiable. The type of machine learning model constituting the feature extractor 501 is not particularly limited and may be selected appropriately depending on the embodiment. For example, a neural network may be used as the machine learning model constituting the feature extractor 501. The structure of the neural network may be determined appropriately depending on the embodiment. When a neural network is used for the feature extractor 501, examples of parameters include the weight of the connections between each node (neuron) and the threshold value of each node.

[0039] The number of feature extractors 501 is not particularly limited and may be determined appropriately depending on the embodiment. In one example, as shown in FIG. 2, the feature extractor 501 may be used in common for at least some of the multiple sets included in the input graph. In a simple case, the feature extractor 501 may be used in common for all the sets. In this case, the pre-processing module 50 may include a single feature extractor 501. In another example, a separate feature extractor 501 may be provided for each set. In this case, the pre-processing module 50 may include multiple feature extractors 501.

[0040] The selection module 503 according to this embodiment is configured to select one or more edges extending from each element to other elements of other sets based on the feature values ​​of each element calculated for each set by the feature extractor 501. Furthermore, the selection module 503 is configured to generate, for each set, graph information indicating the calculated feature values ​​of each element and one or more edges selected for each element. In the example of FIG. 2 , the input graph includes set A and set B, and the feature extractor 501 calculates feature values ​​(A, B) of each element of each set (A, B). The selection module 503 selects, for each element, one or more edges extending from each element based on the feature values ​​(A, B) of each element of each set (A, B), thereby deriving graph information (A, B) for each set (A, B). The data format of the graph information is not particularly limited as long as it indicates the feature values ​​of each element and the results of the edge selection. In one example, the graph information for each set is derived by combining the feature values ​​of each element and the information indicating the branch selection results (for example, by performing a cat operation). It is okay to do so.

[0041] In one example of branch selection, the selection module 503 may select one or more branches extending from each element as a starting point, regardless of whether branches are previously assigned between the elements. In other words, as a result of the branch selection by the selection module 503, a branch may be established between elements to which no branch is previously assigned in the input graph. For example, candidate elements (hereinafter also referred to as "candidate elements") to which a branch is to be extended from a target element belonging to a target set may be all elements belonging to sets other than the target set. The selection module 503 may be configured to select, from all of the elements, candidates to be combined with the target element to establish a branch (i.e., select a combination of elements to establish a branch between each set). However, the form of branch selection is not limited to this example. In another example, the selection module 503 may select one or more branches from previously assigned branches.

[0042] The method of selecting a branch based on the feature amount is not particularly limited as long as it can select a branch useful for inferring a task from the feature amount of each element, and is determined appropriately depending on the embodiment. In one example, selecting one or more branches based on the feature values ​​of each element may include calculating the likelihood between each element and each candidate element that is a candidate for connecting a branch from each element, based on the feature values ​​of each element, and selecting one or more branches based on the likelihood calculated for each candidate element. This allows appropriate selection of branches to be referenced when solving an inference task. Note that the likelihood indicates the degree to which a combination of each element and each candidate element is related to the inference of the task. The likelihood may also be referred to by other names, such as similarity.

[0043] Fig. 3 schematically shows an example of the calculation process of the preprocessing module 50 according to this embodiment. In Fig. 3, two pieces of point cloud data (set A and set B) showing a human body model are given as input graphs, and a situation is assumed in which combinations of corresponding points (elements) between the point cloud data are identified. It is assumed that the number of points (elements) in the point cloud data of set A is N, and the number of points (elements) in the point cloud data of set B is M.

[0044] In the example of FIG. 3 , first, the pre-processing module 50 calculates the feature quantities (A, B) of each element of each set (A, B) using the feature extractor 501. The feature quantities (A, B) of each element may be calculated from attribute values ​​(e.g., coordinates, etc.) of points in the point cloud data. Next, the pre-processing module 50 calculates the likelihood between each element of each set (A, B) and each candidate element of the other set (B, A) using the selection module 503. The method of calculating the likelihood may be selected appropriately depending on the embodiment. In this embodiment, the likelihood of each element may be calculated by a predetermined calculation. In one example, the selection module 503 may calculate the likelihood using the following calculation formula 1.

[0045]

number

[0046] Next, the preprocessing module 50 selects one or more branches for each element of each set (A, B) according to the calculated likelihood using the selection module 503. The method for selecting the branches may be appropriately selected depending on the embodiment. In one example, the selection module 503 may select k branches for each element. In this case, any algorithm such as the k-nearest neighbor method may be used as the method for selecting the k branches. In another example, a maximum number K of branches to be selected for each set (A, B) may be set, and the selection module 503 may select at least one or more branches (a variable number) for each element. For example, when prioritizing candidates with high likelihood, the selection module 503 may select the candidate with the highest likelihood for each element (N candidates in the case of set A), and then select the second and subsequent candidates (KN candidates) in descending order of likelihood.

[0047] Then, the preprocessing module 50 generates graph information (A, B) for each set (A, B) by the selection module 503, which indicates the calculated feature amount (A, B) of each element and one or more branches selected for each element. The graph information A may be appropriately configured to indicate the feature amount A of each element of set A and the branch selection result for each element of set A, and the graph information B may be appropriately configured to indicate the feature amount B of each element of set B and the branch selection result for each element of set B. Note that the branch selection result may depend on, for example, the type of graph, the branch selection method, etc. In the example of FIG. 3, selected branches are indicated by solid lines, and unselected branches are indicated by dotted lines. In this example, For example, for the first element of set A, branches between the first element of set B and each of the M elements are selected. In contrast, for the first element of set B, branches between the first element of set A and the M element of set A are not selected, but branches between the second element of set A and the M element of set A are selected. In this way, in some cases, the branch selection results may differ between set A and set B.

[0048] Returning to FIGS. 1 and 2 , the graph inference module 55 is configured to be differentiable and to infer a task solution for the input graph from the graph information of each generated set. The graph inference module 55 may be configured by a machine learning model having one or more parameters for executing a computation process to infer a task solution from the graph information, the parameters being adjusted by machine learning. The type of machine learning model constituting the graph inference module 55 is not particularly limited and may be selected appropriately depending on the embodiment. For example, a neural network may be used as the machine learning model constituting the graph inference module 55. The structure of the neural network may be determined appropriately depending on the embodiment. When a neural network is used for the graph inference module 55, examples of parameters include the weight of the connections between each node and the threshold value of each node.

[0049] In one example, the graph inference module 55 is implemented as a graph inference algorithm based on the method described in Reference 1 (Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, Xiaowei Zhou, "LoFTR: Detector-Free Local Feature Matching"). with Transformers” [online], [Retrieved January 11, 2022], Internet<URL: https: / / arxiv.org / abs / 2104.00680> ) proposed in It may be done.

[0050] In another example, the graph inference module 55 may be configured by WeaveNet proposed in Non-Patent Document 3. In this case, the graph inference module 55 may include a feature weaving network and an inferer. The feature weaving network may be configured to include multiple feature weaving layers, and to accept input of graph information and output feature information. The inferer may be configured to infer a solution to a task from the feature information output by the feature weaving network. Each feature weaving layer may accept input of L (L+1)-th order input tensors (L is a natural number greater than or equal to 2). The i-th input tensor among the L input tensors may be configured such that the i-th vertex (i-th element belonging to the i-th set) belonging to the i-th set is arranged as an element on the first axis, the vertices belonging to the (j-1)-th set are arranged as elements on the j-th axis from the second axis to the L-th axis, starting from the i-th vertex and circulating, and the feature quantities related to the branches flowing out from the i-th vertex belonging to the i-th set to combinations of vertices belonging to each of the (L-1) other sets are arranged on the (L+1)-th axis. Each feature weaving layer may include an encoder. Each feature weaving layer may be configured to, for an i-th input tensor from the first to the Lth input tensor, fix the (L+1)-th axis of each of the (L-1) other input tensors other than the i-th input tensor, rotate the axes other than the (L+1)-th axis of each of the other input tensors so that they align with the axes of the i-th input tensor, and concatenate features of each element of the i-th input tensor and each of the rotated other input tensors to generate L (L+1)-th-order concatenated tensors, and to divide each of the generated L concatenated tensors into elements along the first axis and input them to an encoder to perform an encoder operation, thereby generating L (L+1)-th-order output tensors corresponding to each of the L input tensors. The encoder may be configured to derive relative features of each element from the features of all input elements. The first feature weaving layer of the multiple feature weaving layers may be configured to receive the L input tensors directly from graph information of each set or indirectly via a predetermined operation. The feature information may be composed of L output tensors output from the last feature weaving layer among the multiple feature weaving layers.By adopting WeaveNet in the graph inference module 55, the accuracy of inferring a solution to a task can be improved by deepening the hierarchy of the feature weaving network, even for complex tasks.

[0051] Machine learning in the model generation device 1 is performed by inputting each training graph 30 as an input graph into the preprocessing module 50, and training the inference model 5 so that the inference result obtained from the graph inference module 55 matches the correct answer (true value) of the task for each training graph 30. During this machine learning, the inference model 5 may be trained consistently from the graph inference module 55 to the feature extractor 501. Meanwhile, inferring a solution to a task for the target graph 221 in the inference device 2 is performed by inputting the target graph 221 as an input graph into the preprocessing module 50, and obtaining the result of inferring the solution to the task from the graph inference module 55.

[0052] (Features) As described above, in the inference model 5 according to this embodiment, the preprocessing module 50 calculates the feature values ​​of each element of the input graph, and based on the calculated feature values ​​of each element, it is possible to select edges to be referenced when solving an inference task. As a result, even if the number of elements included in the data increases, the number of edges to be referenced in the inference process can be reduced by edge selection by the preprocessing module 50. Therefore, according to this embodiment, it is possible to improve the efficiency of processing when solving an inference task for data having a graph structure. The model generation device 1 according to this embodiment can generate a trained inference model 5 that can efficiently perform inference processing. The inference device 2 according to this embodiment can efficiently solve an inference task for the target graph 221 by using the generated trained inference model 5.

[0053] A simple method would be to select branches based on rules to improve the efficiency of the inference process. However, with this method, the rules for selecting branches are not necessarily optimal for the inference process, and the performance of the inference process is limited by the rules for selecting the branches, which could result in a deterioration in accuracy. In contrast, in this embodiment, in the machine learning of the inference model 5, the graph inference module 55 to the feature extractor 501 can be consistently trained. This allows the performance of the feature extractor 501 to be optimized for the inference process, thereby preventing a deterioration in inference accuracy. Therefore, according to this embodiment, the efficiency of the inference process can be improved while ensuring the accuracy of the inference.

[0054] In addition, if there is diversity in the variation of the feature values, that diversity may lead to a deterioration in inference accuracy. In contrast, in this embodiment, the selection of branches by the selection module 503 can suppress the diversity of the variation of the feature values ​​processed by the graph inference module 55. As a result, an improvement in inference accuracy can be expected.

[0055] §2 Configuration example [Hardware configuration] <Model generation device> Fig. 4 schematically illustrates an example of the hardware configuration of the model generation device 1 according to this embodiment. As shown in Fig. 4, the model generation device 1 according to this embodiment is a computer to which a control unit 11, a storage unit 12, a communication interface 13, an external interface 14, an input device 15, an output device 16, and a drive 17 are electrically connected.

[0056] The control unit 11 includes a CPU (Central Processing Unit) which is a hardware processor, The system includes RAM (Random Access Memory), ROM (Read Only Memory), etc., and is configured to execute information processing based on programs and various data. The control unit 11 (CPU) is an example of a processor resource. The storage unit 12 is an example of a memory resource, and is configured, for example, with a hard disk drive, a solid state drive, etc. In this embodiment, the storage unit 12 stores various information such as a model generation program 81, multiple training graphs 30, and learning result data 125.

[0057] The model generation program 81 is a program for causing the model generation device 1 to execute machine learning information processing (FIG. 8) described below to generate a trained inference model 5. The model generation program 81 includes a series of instructions for the information processing. Multiple training graphs 30 are used for machine learning of the inference model 5. The learning result data 125 indicates information about the trained inference model 5 generated by performing machine learning. In this embodiment, the learning result data 125 is generated as a result of executing the model generation program 81.

[0058] The communication interface 13 is, for example, a wired LAN (Local Area Network) module, a wireless LAN module, or the like, and is an interface for performing wired or wireless communication via a network. The model generation device 1 can use the communication interface 13 to perform data communication with other information processing devices via a network. The external interface 14 is, for example, a USB (Universal Serial Bus) port, a dedicated port, or the like, and is an interface for connecting to an external device. The type and number of external interfaces 14 may be selected arbitrarily. The training graph 30 may be obtained by a sensor such as a camera. Alternatively, the training graph 30 may be generated by another computer. In these cases, the model generation device 1 may be connected to the sensor or the other computer via at least one of the communication interface 13 and the external interface 14.

[0059] The input device 15 is a device for inputting, for example, a mouse, a keyboard, etc. The output device 16 is a device for outputting, for example, a display, a speaker, etc. An operator can operate the model generation device 1 by using the input device 15 and the output device 16. The training graph 30 may be obtained by input via the input device 15. The input device 15 and the output device 16 may be integrated into one device, for example, a touch panel display, etc.

[0060] The drive 17 is, for example, a CD drive, a DVD drive, or the like, and is a drive device for reading various information, such as programs, stored in a storage medium 91. The storage medium 91 is a medium that stores information, such as programs, electrically, magnetically, optically, mechanically, or chemically, so that a computer or other device, machine, or the like can read the stored information. At least one of the model generation program 81 and the multiple training graphs 30 may be stored in the storage medium 91. The model generation device 1 may acquire at least one of the model generation program 81 and the multiple training graphs 30 from the storage medium 91. Here, FIG. 4 illustrates a disk-type storage medium, such as a CD or a DVD, as an example of the storage medium 91. However, the type of the storage medium 91 is not limited to a disk-type storage medium and may be other than a disk-type storage medium. Examples of storage media other than a disk-type storage medium include semiconductor memories, such as flash memories. The type of the drive 17 may be selected arbitrarily depending on the type of the storage medium 91.

[0061] It should be noted that, with regard to the specific hardware configuration of the model generating device 1, components can be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit 11 may include multiple hardware processors. The hardware processors may be microprocessors, FPGAs (field-programmable gate arrays), DSPs (digital signal processors), etc. The storage unit 12 may be configured with RAM and ROM included in the control unit 11. At least one of the communication interface 13, the external interface 14, the input device 15, the output device 16, and the drive 17 may be omitted. The model generation device 1 may be configured with multiple computers. In this case, the hardware configurations of the computers may or may not be the same. Furthermore, the model generation device 1 may be an information processing device designed specifically for the service to be provided, as well as a general-purpose server device, a PC (Personal Computer), etc.

[0062] <Inference device> Fig. 5 shows a schematic example of a hardware configuration of the inference device 2 according to this embodiment. As shown in Fig. 5, the inference device 2 according to this embodiment is a computer in which a control unit 21, a storage unit 22, a communication interface 23, an external interface 24, an input device 25, an output device 26, and a drive 27 are electrically connected.

[0063] The control unit 21 to the drive 27 and the storage medium 92 of the inference device 2 may be configured similarly to the control unit 11 to the drive 17 and the storage medium 91 of the model generation device 1, respectively. The control unit 21 includes a hardware processor such as a CPU, RAM, and ROM, and is configured to execute various information processes based on programs and data. The storage unit 22 is configured, for example, with a hard disk drive or a solid state drive. In this embodiment, the storage unit 22 stores various information such as an inference program 82 and learning result data 125.

[0064] The inference program 82 is a program for causing the inference device 2 to execute information processing (FIG. 9) described below, which performs an inference task using a trained inference model 5. The inference program 82 includes a series of instructions for the information processing. At least one of the inference program 82 and the learning result data 125 may be stored in a storage medium 92. Furthermore, the inference device 2 may acquire at least one of the inference program 82 and the learning result data 125 from the storage medium 92.

[0065] Similar to the training graph 30, the object graph 221 may be obtained by a sensor such as a camera. Alternatively, the object graph 221 may be generated by another computer. In these cases, the inference device 2 may be connected to the sensor or the other computer via at least one of the communication interface 23 and the external interface 24. The object graph 221 may be obtained by input via the input device 25.

[0066] Note that, with regard to the specific hardware configuration of the inference device 2, components can be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit 21 may include multiple hardware processors. The hardware processor may be configured with a microprocessor, FPGA, DSP, etc. The storage unit 22 may be configured with RAM and ROM included in the control unit 21. At least one of the communication interface 23, the external interface 24, the input device 25, the output device 26, and the drive 27 may be omitted. The inference device 2 may be configured with multiple computers. In this case, the hardware configurations of the computers may or may not be the same. Furthermore, the inference device 2 may be an information processing device designed specifically for the service provided, as well as a general-purpose server device, a general-purpose PC, a tablet PC, a terminal device, etc.

[0067] [Software configuration] <Model generation device> FIG. 6 schematically illustrates an example of the software configuration of the model generation device 1 according to this embodiment. The control unit 11 of the model generation device 1 loads the model generation program 81 stored in the storage unit 12 into the RAM. The control unit 11 then uses the CPU to interpret and execute instructions included in the model generation program 81 loaded into the RAM, thereby controlling each component. As a result, as shown in FIG. 6, the model generation device 1 according to this embodiment operates as a computer including an acquisition unit 111, a learning processing unit 112, and a storage processing unit 113 as software modules. That is, in this embodiment, each software module of the model generation device 1 is realized by the control unit 11 (CPU).

[0068] The acquisition unit 111 is configured to acquire a plurality of training graphs 30. The learning processing unit 112 is configured to use the acquired plurality of training graphs 30 to perform machine learning of the inference model 5. The inference model 5 includes a preprocessing module 50 and a graph inference module 55. The machine learning is configured by inputting each training graph 30 as an input graph into the preprocessing module 50, and training the inference model 5 so that the inference result obtained by the graph inference module 55 matches the correct answer (true value) of the task for each training graph 30.

[0069] Training the inference model 5 involves adjusting (optimizing) the values ​​of parameters included in the inference model 5 so as to derive an output (inference result) that matches each training graph 30 from each training graph 30. The machine learning method may be selected appropriately depending on the type of machine learning model to be adopted, etc. Examples of the machine learning method that may be adopted include backpropagation, solving an optimization problem, etc.

[0070] In this embodiment, a neural network may be used for the inference model 5 (the feature extractor 501 and the graph inference module 55). In this case, each training graph 30 is input as an input graph to the preprocessing module 50, and the preprocessing module 50 and the graph inference module 55 perform forward calculation processing, thereby obtaining an inference result for each training graph 30 as an output of the graph inference module 55. The learning processing unit 112 is configured to adjust parameters of the inference model 5 (for example, parameters of the feature extractor 501 and the graph inference module 55) in the machine learning process so as to reduce the error between the inference result obtained for each training graph 30 and the correct answer.

[0071] The correct answer (true value) for each training graph 30 may be given by a predetermined rule, such as not having a blocking pair in the case of a stable matching problem, satisfying a predetermined criterion, or having points that have a corresponding relationship. Alternatively, a correct answer label (teacher signal) may be associated with each acquired training graph 30. Each correct answer label may be configured to indicate the correct answer (true value) of the inference task for the corresponding training graph 30.

[0072] The storage processing unit 113 is configured to generate information about the trained inference model 5 generated by the machine learning as learning result data 125 and store the generated learning result data 125 in a predetermined storage area. As long as the trained inference model 5 can be reproduced, the configuration of the learning result data 125 is not particularly limited and may be determined appropriately depending on the embodiment. As an example, the learning result data 125 may include information indicating the values ​​of each parameter obtained by adjusting the machine learning. In some cases, the learning result data 125 may include information indicating the structure of the inference model 5. The structure may be specified by, for example, the number of layers, the type of each layer, the number of nodes included in each layer, the connection relationships between nodes in adjacent layers, etc.

[0073] <Inference device> 7 schematically illustrates an example of the software configuration of the inference device 2 according to this embodiment. The control unit 21 of the inference device 2 loads an inference program 82 stored in the storage unit 22 onto the RAM. The control unit 21 then uses the CPU to interpret and execute instructions included in the inference program 82 loaded onto the RAM, thereby controlling each component. As shown in FIG. 7, the inference device 2 according to this embodiment thereby operates as a computer having an acquisition unit 211, an inference unit 212, and an output unit 213 as software modules. That is, in this embodiment, each software module of the inference device 2 is also realized by the control unit 21 (CPU), similar to the model generation device 1.

[0074] The acquisition unit 211 is configured to acquire a target graph 221. The inference unit 212 holds the learning result data 125, thereby providing a trained inference model 5. 12 is configured to infer a task solution for the obtained object graph 221 using the trained inference model 5. Inferring a task solution for the object graph 221 is configured by inputting the object graph 221 as an input graph to the preprocessing module 50 and obtaining a result of inferring a task solution from the graph inference module 55. The output unit 213 is configured to output information regarding the result of inferring a task solution.

[0075] <Other> Each software module of the model generation device 1 and the inference device 2 will be described in detail in an operation example described later. Note that in this embodiment, an example is described in which each software module of the model generation device 1 and the inference device 2 is implemented by a general-purpose CPU. However, some or all of the above software modules may be implemented by one or more dedicated processors (e.g., graphics processing units). Each of the above modules may be implemented as a hardware module. Furthermore, with regard to the software configuration of each of the model generation device 1 and the inference device 2, software modules may be omitted, replaced, or added as appropriate depending on the embodiment.

[0076] §3 Example of operation [Model generation device] 8 is a flowchart showing an example of a processing procedure for machine learning by the model generation device 1 according to this embodiment. The processing procedure of the model generation device 1 described below is an example of a model generation method. However, the processing procedure of the model generation device 1 described below is merely an example, and each step may be changed as much as possible. Furthermore, steps may be omitted, replaced, or added to the following processing procedure as appropriate depending on the embodiment.

[0077] (Step S101) In step S101, the control unit 11 operates as the acquisition unit 111 and acquires a plurality of training graphs 30.

[0078] Each training graph 30 may be generated as appropriate depending on the inference capability of the task to be acquired by the inference model 5. Conditions for solving the task may be given as appropriate, and each training graph 30 may be generated from the given conditions. Each training graph 30 may be obtained from existing data representing an existing transportation network, an existing communication network, etc. Each training graph 30 may be obtained from image data. The image data may be obtained by a camera or may be generated as appropriate by a computer. Furthermore, each training graph 30 may be obtained from point cloud data representing a two-dimensional or three-dimensional model.

[0079] The data format of each training graph 30 is not particularly limited and may be selected appropriately depending on the embodiment. In one example, each training graph 30 may be configured using an adjacency list or an adjacency matrix. In another example, each training graph 30 may be configured using data formats other than an adjacency list or an adjacency matrix (e.g., image data, point cloud data, etc.). As another example, each training graph 30 may be configured using image data. In this case, each pixel may correspond to an element (vertex), and the relationship between pixels may correspond to a branch. As yet another example, each training graph 30 may be configured using point cloud data. In this case, each point included in the point cloud data may correspond to an element.

[0080] A correct answer (true value) for each training graph 30 may be given as appropriate. In one example, the correct answer (true value) may be given according to a predetermined rule. In another example, the correct answer (true value) may be indicated by a correct answer label (teacher signal). In this case, a correct answer label corresponding to each training graph 30 may be generated as appropriate, and the generated correct answer label may be associated with each training graph 30. As a result, each training graph 30 is stored in the form of a dataset associated with a correct answer label. may be generated.

[0081] Each training graph 30 may be generated automatically by the operation of a computer, or may be generated manually with at least a partial operation by an operator. Furthermore, each training graph 30 may be generated by the model generation device 1, or by a computer other than the model generation device 1. That is, the control unit 11 may generate each training graph 30 automatically or manually by an operator's operation via the input device 15. Alternatively, the control unit 11 may acquire each training graph 30 generated by another computer, for example, via a network, a storage medium 91, or the like. Some of the multiple training graphs 30 may be generated by the model generation device 1, and the others may be generated by one or more other computers. At least one of the multiple training graphs 30 may be generated by, for example, a generative model configured by a machine learning model (e.g., a generative model included in a generative adversarial network).

[0082] The number of training graphs 30 to be acquired is not particularly limited and may be determined appropriately depending on the embodiment so that machine learning can be performed. After acquiring a plurality of training graphs 30, the control unit 11 proceeds to the next step S102.

[0083] (Step S102) In step S102, the control unit 11 operates as the learning processing unit 112 and performs machine learning of the inference model 5 using the acquired multiple training graphs 30.

[0084] As an example of machine learning processing, first, the control unit 11 performs initial settings for the inference model 5 that is the target of machine learning processing. The structure and initial parameter values ​​of the inference model 5 may be provided by a template or may be determined by input by an operator. Furthermore, when additional learning or re-learning is performed, the control unit 11 may perform initial settings for the inference model 5 based on learning result data obtained by past machine learning.

[0085] Next, the control unit 11 trains the inference model 5 by machine learning so that the result of inferring the solution to the task for each training graph 30 matches the correct answer (true value) (i.e., adjusts the parameter values ​​of the inference model 5). For this training process, a stochastic gradient descent method, a mini-batch gradient descent method, or the like may be used.

[0086] As an example of the training process, first, the control unit 11 inputs each training graph 30 into the inference model 5 and executes forward calculation processing. In one example, the control unit 11 inputs information indicating features related to each element belonging to each set of each training graph 30 (e.g., features of the branches connecting to each element, features of each element itself) to the feature extractor 501 and executes calculation processing of the feature extractor 501. As a result, the control unit 11 calculates the feature amount of each element for each set. Next, the control unit 11 inputs the feature amount of each element calculated for each set to the selection module 503 and executes calculation processing of the selection module 503. In the selection module 503, the control unit 11 selects one or more branches extending from each element as a starting point based on the feature amount of each element. In this embodiment, the control unit 11 may calculate the likelihood between each element and each candidate element from the feature amount and select one or more branches for each element according to the calculated likelihood. The control unit 11 generates graph information for each set that indicates the calculated feature amount of each element and the selected one or more branches. Then, the control unit 11 inputs the graph information generated for each set to the graph inference module 55 and executes the calculation process of the graph inference module 55. Through this series of forward calculation processes, the control unit 11 obtains from the graph inference module 55 the inference result of the task solution for each training graph 30.

[0087] Next, the control unit 11 calculates the error between the obtained inference result and the corresponding correct answer. A loss function may be used to calculate the error. The loss function may be, for example, a function that depends on the task, the format of the correct answer, etc. The control unit 11 calculates the gradient of the calculated error. The control unit 11 calculates the error of the parameter values ​​of the inference model 5 sequentially from the output side using the gradient of the calculated error by the error backpropagation method. The control unit 11 updates the value of each parameter of the inference model 5 based on each calculated error. In one example, the parameters of the inference model 5 include each parameter of the graph inference module 55 and the preprocessing module 50 (feature extractor 501), and the control unit 11 may update the value of each parameter consistently from the graph inference module 55 to the feature extractor 501. The degree to which the value of each parameter is updated may be adjusted by a learning rate. The learning rate may be specified by an operator or may be set as a setting value within the program.

[0088] The control unit 11 adjusts the value of each parameter through the series of update processes described above so that the sum of calculated errors for each training graph 30 is reduced. For example, the control unit 11 may repeat the adjustment of the value of each parameter through the series of update processes described above until a predetermined condition is met, such as a specified number of executions or the sum of calculated errors becoming equal to or less than a threshold. As a result of this machine learning process, the control unit 11 can generate a trained inference model 5 that has acquired the ability to perform a desired inference task according to the training graph 30 used (i.e., to infer a solution to the task for a given graph). When the machine learning process is complete, the control unit 11 proceeds to the next step S103.

[0089] (Step S103) In step S103, the control unit 11 operates as a storage processing unit 113 and generates information about the trained inference model 5 generated by machine learning as learning result data 125. Then, the control unit 11 stores the generated learning result data 125 in a predetermined storage area.

[0090] The predetermined storage area may be, for example, RAM in the control unit 11, the storage unit 12, an external storage device, a storage medium, or a combination thereof. The storage medium may be, for example, a CD, a DVD, or the like, and the control unit 11 may store the learning result data 125 in the storage medium via the drive 17. The external storage device may be, for example, a data server such as a network-attached storage (NAS). In this case, the control unit 11 may use the communication interface 13 to store the learning result data 125 in the data server via a network. The external storage device may also be, for example, an external storage device connected to the model generation device 1 via the external interface 14.

[0091] When the storage of the learning result data 125 is completed, the control unit 11 ends the processing procedure of the model generating device 1 according to this operation example.

[0092] The generated learning result data 125 may be provided to the inference device 2 at any timing. For example, the control unit 11 may transfer the learning result data 125 to the inference device 2 as part of the processing of step S103 or separately from the processing of step S103. The inference device 2 may acquire the learning result data 125 by receiving this transfer. Also, for example, the inference device 2 may acquire the learning result data 125 by accessing the model generation device 1 or a data server via a network using the communication interface 23. Also, for example, the inference device 2 may acquire the learning result data 125 via the storage medium 92. Also, for example, the learning result data 125 may be pre-installed in the inference device 2.

[0093] Furthermore, the control unit 11 may update or generate new learning result data 125 by periodically or irregularly repeating the processes of steps S101 to S103. During this repetition, at least a part of the training graph 30 used for machine learning may be changed, modified, added, deleted, etc. as appropriate. Then, the control unit 11 may update or generate new learning result data 125. The learning result data 125 held by the inference device 2 may be updated by providing the result data 125 to the inference device 2 in any manner.

[0094] [Inference device] 9 is a flowchart showing an example of a processing procedure for performing an inference task by the inference device 2 according to this embodiment. The processing procedure of the inference device 2 described below is an example of an inference method. However, the processing procedure of the inference device 2 described below is merely an example, and each step may be modified as much as possible. Furthermore, steps may be omitted, replaced, or added to the following processing procedure as appropriate depending on the embodiment.

[0095] (Step S201) In step S201, the control unit 21 operates as the acquisition unit 211 and acquires a target graph 221 that is the target of the inference task. The configuration of the target graph 221 is the same as that of the training graph 30. The data format of the target graph 221 may be selected appropriately depending on the embodiment. The target graph 221 may be generated from conditions input via the input device 25, for example. The target graph 221 may be obtained from existing data indicating, for example, an existing transportation network, an existing communication network, etc. Alternatively, the target graph 221 may be obtained from, for example, image data, point cloud data, etc. In one example, the control unit 21 may directly acquire the target graph 221. In another example, the control unit 21 may indirectly acquire the target graph 221 via, for example, a network, a sensor, another computer, a storage medium 92, etc. Upon acquiring the target graph 221, the control unit 21 proceeds to the next step S202.

[0096] (Step S202) In step S202, the control unit 21 operates as the inference unit 212 and sets the trained inference model 5 by referring to the learning result data 125. Then, the control unit 21 uses the trained inference model 5 to infer a task solution for the acquired target graph 221. This inference calculation process may be similar to the forward calculation process in the above-mentioned machine learning training process. The control unit 21 inputs the target graph 221 into the trained inference model 5 and executes the forward calculation process of the trained inference model 5. As a result of executing this calculation process, the control unit 21 can acquire the result of inferring the task solution for the target graph 221 from the graph inference module 55. Upon acquiring the inference result, the control unit 21 proceeds to the next step S203.

[0097] (Step S203) In step S203, the control unit 21 operates as the output unit 213 and outputs information related to the inference result.

[0098] The output destination and the content of the information to be output may be determined appropriately depending on the embodiment. For example, the control unit 21 may output the inference result obtained in step S202 directly to the output device 26 or an output device of another computer. The control unit 21 may also perform some kind of information processing based on the obtained inference result. The control unit 21 may then output the result of the information processing as information related to the inference result. The output of the result of the information processing may include controlling the operation of the controlled device in accordance with the inference result. The output destination may be, for example, the output device 26, an output device of another computer, the controlled device, etc.

[0099] When the output of information related to the inference result is completed, the control unit 21 ends the processing procedure of the inference device 2 according to this operation example. The control unit 21 may continuously repeat the series of information processing steps S201 to S203. The timing of the repetition may be determined appropriately depending on the embodiment. In this way, the inference device 2 continuously repeats the above inference task. The method may be configured to return the

[0100] [Features] As described above, according to the inference model 5 of this embodiment, the number of branches referenced in the inference process can be reduced by branch selection by the preprocessing module 50. When solving an inference task for data having a graph structure, the efficiency of the process can be improved. In the model generation device 1, a trained inference model 5 capable of efficiently performing the inference process can be generated by the processes of steps S101 and S102. In the machine learning of step S102, consistent training from the graph inference module 55 to the feature extractor 501 can be performed, thereby ensuring the accuracy of the inference and generating a trained inference model 5 capable of efficiently performing the inference process. In the inference device 2, the processes of steps S201 to S203 can efficiently solve the inference task for the target graph 221. Furthermore, in the process of step S202, branch selection by the selection module 503 can reduce the diversity of variance in the features processed by the graph inference module 55. As a result, improved inference accuracy can be expected.

[0101] §4 Variations Although the embodiments of the present invention have been described above in detail, the above description is merely illustrative of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. For example, the following modifications are possible. Note that, in the following, the same reference numerals are used for components similar to those in the above embodiment, and descriptions of the same points as those in the above embodiment are omitted where appropriate. The following modifications can be combined as appropriate.

[0102] <4.1> The inference system 100 according to the above embodiment may be applied to solving various tasks that can be expressed by graphs. The type of graph (training graph 30, target graph 221) may be selected appropriately depending on the task. The graph may be, for example, a directed bipartite graph, an undirected bipartite graph, a general directed graph, a general undirected graph, a vertex feature graph, a hypergraph, etc. The task may be, for example, two-sided matching, one-sided matching, event estimation (including prediction), graph feature estimation, search within a graph, vertex categorization (graph division), estimation of relationships between vertices, etc. Below, specific examples are given that limit the application scenario.

[0103] (A) Situations where directed bipartite graphs are used FIG. 10 schematically illustrates an example of an application scenario of an inference system 100A according to a first specific example. The first specific example is an example in which the above-described embodiment is applied to a scenario in which a directed bipartite graph is used as a graph representing a given condition (a target for solving a task). The inference system 100A according to the first specific example includes a model generation device 1 and an inference device 2A. The inference device 2A is an example of the inference device 2 described above.

[0104] In the first specific example, each training graph 30A and object graph 221A (input graph) are directed bipartite graphs 70A. The multiple elements (vertices) included in the directed bipartite graph 70A are divided into two subsets (71A, 72A) so that there are no edges between elements within each subset (71A, 72A). Edges from elements belonging to one of the two subsets (71A, 72A) to elements belonging to the other subset, and edges from elements belonging to the other subset to elements belonging to the one subset, are individually defined. The multiple sets of the input graph are composed of two subsets (71A, 72A) of the directed bipartite graph 70A. Each training graph 30A and object graph 221A are examples of each training graph 30 and object graph 221 in the above embodiment. The number of elements in the directed bipartite graph 70A and the presence or absence of directed edges between elements may be appropriately determined to appropriately represent the target of solving a task.

[0105] As long as the task can be expressed by a directed bipartite graph, its content is not particularly limited and may be selected appropriately depending on the embodiment. In one example, the task may be a two-sided matching. That is, the task may be a matching task between two parties, in which the optimal pairing of objects belonging to each party is determined. In this case, two subsets (71A, 72A) of the directed bipartite graph 70A correspond to the two parties in the matching task. The elements belonging to each subset (71A, 72A) of the directed bipartite graph 70A correspond to the objects belonging to each party.

[0106] The characteristics of the directed branch flowing from an element belonging to one subset to an element belonging to the other subset correspond to the degree of desirability from an object belonging to one party to an object belonging to the other party. Similarly, the characteristics of the directed branch flowing from an element belonging to the other subset to an element belonging to one subset correspond to the degree of desirability from an object belonging to the other party to an object belonging to one party. In other words, the characteristics of the directed branch flowing from each element correspond to the degree of desirability from the object corresponding to each element to an object belonging to another party (a party other than the party to which the corresponding object belongs).

[0107] As long as it can indicate the degree of desire to be paired (matched), the format of the degree of desire need not be particularly limited and may be selected appropriately depending on the embodiment. The degree of desire may be expressed, for example, as a preference order, a score, etc. The score indicates the degree of desire to be paired as a numerical value. The score may be assigned appropriately. In one example, after each subject provides a list of preference orders, a score may be obtained by numerically converting the preference orders indicated in each subject's list using a predetermined calculation, and the obtained score may be used as the degree of desire.

[0108] The matching target may be selected appropriately depending on the embodiment. The matching target may be, for example, male / female, roommate, job seeker / employer, employee / placement, patient / doctor, energy supplier / recipient, etc. In addition, since the task is matching, the inference device 2A may be read as a matching device.

[0109] The graph inference module 55 may be configured to infer a matching result from the graph information of each subset (71A, 72A). A correct answer in machine learning may be appropriately given so as to obtain a matching result that meets a desired criterion.

[0110] In the first specific example, during machine learning by the model generation device 1 and during inference processing by the inference device 2A, the directed bipartite graph 70A is treated as an input graph. Calculating the feature values ​​of each element belonging to each set is performed by calculating the feature values ​​of each element from the features of the edges extending from each element belonging to each subset (71A, 72A) of the directed bipartite graph 70A as the starting point (in other words, the directed edges emanating from each element). That is, the feature extractor 501 is configured to calculate the feature values ​​of each element for each subset (71A, 72A) of the directed bipartite graph 70A from the features of the directed edges emanating from each element. Except for these points, the configuration of the first specific example may be the same as that of the above-described embodiment.

[0111] (Model generation device) In the first specific example, the model generation device 1 can generate a trained inference model 5 that has acquired the ability to infer solutions to tasks for the directed bipartite graph 70A using a processing procedure similar to that of the above embodiment.

[0112] That is, in step S101, the control unit 11 acquires a plurality of training graphs 30A. Each training graph 30A is composed of a directed bipartite graph 70A. The elements (vertices) and edges in each training graph 30A are set appropriately so as to appropriately express the conditions of the training subject. In step S102, the control unit 11 performs machine learning of the inference model 5 using the acquired multiple training graphs 30A. This machine learning makes it possible to generate a trained inference model 5 that has acquired the ability to infer a solution to a task for the directed bipartite graph 70A. The task may be the above-mentioned matching task. In step S103, the control unit 11 generates learning result data 125 indicating the generated trained inference model 5 and stores the generated learning result data 125 in a predetermined storage area. The learning result data 125 may be provided to the inference device 2A at any timing.

[0113] (Inference device) The hardware and software configurations of the inference device 2A may be similar to those of the inference device 2 according to the above embodiment. In the first specific example, the inference device 2A can infer a solution to a task for directed bipartite graph 70A using the same processing procedure as that of the inference device 2 described above.

[0114] That is, in step S201, the control unit of the inference device 2A operates as an acquisition unit and acquires the target graph 221A. The target graph 221A is composed of a directed bipartite graph 70A. The elements (vertices) and edges in the target graph 221A may be set appropriately to appropriately express the conditions of the inference target. In step S202, the control unit operates as an inference unit and uses the trained inference model 5 to infer a solution to the task for the acquired target graph 221A. If the trained inference model 5 has acquired the ability to perform the above matching task, the control unit can obtain the matching inference result.

[0115] In step S203, the control unit operates as an output unit and outputs information related to the inference result. As an example, when the control unit obtains the inference result of the above matching, the control unit may output the obtained inference result as is to the output device. As a result, the inference device 2A may prompt the operator whether or not to adopt the inference result. As another example, when the inference result of the above matching is obtained, the control unit may establish (confirm) at least some of the matching of the inference result. In this case, matching targets (elements) may be added at any timing, and the inference device 2A may repeatedly perform matching between free targets (elements). Targets (elements) for which matching is established by performing a matching task may be excluded from targets of subsequent matching tasks. Furthermore, established matching may be dissolved at any timing, and targets (elements) for which matching is dissolved may be added to targets of subsequent matching tasks. As a result, the inference device 2A may be configured to perform matching online in real time.

[0116] (Features) According to the first specific example, it is possible to improve the efficiency of inference processing in a situation where a given task (e.g., a matching task) is represented by a directed bipartite graph 70A. The model generation device 1 can generate a trained inference model 5 that can efficiently perform the task represented by the directed bipartite graph 70A. By using such a trained inference model 5, the inference device 2A can efficiently infer a solution to the task represented by the directed bipartite graph 70A.

[0117] In the directed bipartite graph 70A, directed edges do not necessarily have to be assigned to all combinations of elements between the two subsets (71A, 72A). That is, there may be combinations of elements to which no directed edges are assigned. In this case, the selection module 503 may select one or more directed edges flowing out from each element, regardless of whether edges have been assigned in advance between the elements. This makes it possible to refer to combinations of elements that are useful for solving a task and infer a solution to the task, even if no directed edges are assigned. As a result, improved inference accuracy can be expected.

[0118] (B) Situations where undirected bipartite graphs are used FIG. 11 schematically illustrates an example of an application scenario of an inference system 100B according to a second specific example. The second specific example is an example in which the above-described embodiment is applied to a scenario in which an undirected bipartite graph is used as a graph representing a given condition (a target for solving a task). The inference system 100B according to the second specific example includes a model generation device 1 and an inference device 2B. The inference device 2B is an example of the inference device 2 described above.

[0119] In the second specific example, each training graph 30B and object graph 221B (input graph) are undirected bipartite graphs 70B. The multiple vertices constituting the undirected bipartite graph 70B are divided into two subsets (71B, 72B) so that there are no edges between vertices within each subset (71B, 72B). Edges from elements belonging to one subset to elements belonging to the other subset, and edges from elements belonging to the other subset to elements belonging to the first subset, are defined without distinction. The multiple sets of the input graph are composed of two subsets (71B, 72B) of the undirected bipartite graph 70B. Each training graph 30B and object graph 221B are examples of each training graph 30 and object graph 221 in the above embodiment. The number of elements in the undirected bipartite graph 70B and the presence or absence of edges between elements may be determined appropriately to appropriately represent the target for solving the task.

[0120] As long as the task can be represented by an undirected bipartite graph, its content may not be particularly limited and may be selected appropriately depending on the embodiment. In one example, the task may be one-sided matching. That is, the task may be a matching task between two parties, in which the optimal pairing of objects belonging to each party is determined. In this case, two subsets (71B, 72B) of the undirected bipartite graph 70B correspond to the two parties in the matching task. Elements (vertices) belonging to each subset (71B, 72B) of the undirected bipartite graph 70B correspond to objects belonging to each party. Characteristics of an edge connecting an element belonging to one of the two subsets (71B, 72B) with an element belonging to the other of the two parties correspond to the cost or reward of pairing an object belonging to one of the two parties with an object belonging to the other party (specifically, two objects corresponding to the two elements connected by the edge).

[0121] The matching target may be selected appropriately depending on the embodiment. The matching target may be, for example, a transport robot / luggage, a room / guest (automatic check-in at a lodging facility), a seat / guest (e.g., seat assignment in public transportation, entertainment facilities, etc.), etc. In another example, matching may be performed to track multiple objects between multiple images (e.g., consecutive frames in a video). In this case, a first image may correspond to one party, and a second image may correspond to another party, and the matching target (vertex) may be an object (or object region) detected in each image. By matching the same object between each image, the object can be tracked.

[0122] The cost indicates the degree to which matching is hindered. The reward indicates the degree to which matching is encouraged. As an example, in a situation where a transport robot and luggage are matched, the distance from the target transport robot to the target luggage may be set as the cost. As another example, in a situation where a seat and a customer are matched, the preference of the target customer (e.g., a preferred position such as a window seat or an aisle seat) may be set as the reward for the target seat. The numerical expression of the cost or reward may be determined appropriately depending on the embodiment. In addition, since the task is matching, the inference device 2B may be read as a matching device.

[0123] In another example, the feature map obtained in a convolutional neural network has height h elements belonging to one subset and width w elements belonging to the other subset. The feature map can be regarded as an undirected bipartite graph. Therefore, the undirected bipartite graph 70B may be configured by such a feature map, and the task may be a predetermined inference for an image. The inference for an image may be, for example, region segmentation or object detection. These predetermined inferences may be performed in various situations, such as extracting an object in an image captured by a vehicle camera or detecting a person in an image captured by a camera installed on a street. The structure of the convolutional neural network is not particularly limited and may be determined appropriately depending on the embodiment. The feature map may be obtained as the output of the convolutional neural network or as a result of calculation during the convolutional neural network.

[0124] The graph inference module 55 may be configured as appropriate to derive an inference result for a solution to a task (e.g., the above-mentioned matching, predetermined inference for an image, etc.) for the undirected bipartite graph 70B from the graph information of each subset (71B, 72B). A correct answer in machine learning may be provided as appropriate depending on the content of the task for the undirected bipartite graph 70B. If the task is the above-mentioned matching task, as in the first specific example, a correct answer in machine learning may be provided as appropriate so as to obtain a matching result that meets a desired standard. If the task is predetermined inference for the above-mentioned image, a correct answer for inference, such as a true value of region segmentation or a true value of an object detection result, may be provided as appropriate to each training graph 30B.

[0125] In the second specific example, during machine learning by the model generation device 1 and during inference processing by the inference device 2B, the undirected bipartite graph 70B is treated as an input graph. Calculating the feature values ​​of each element belonging to each set is configured by calculating the feature values ​​of each element from the features of the edges connecting to each element belonging to each subset (71B, 72B) of the undirected bipartite graph 70B. That is, the feature extractor 501 is configured to calculate the feature values ​​of each element for each subset (71B, 72B) of the undirected bipartite graph 70B from the features of the edges connecting to each element. Except for these points, the configuration of the second specific example may be the same as that of the above embodiment.

[0126] (Model generation device) In the second specific example, the model generation device 1 can generate a trained inference model 5 that has acquired the ability to infer solutions to tasks for undirected bipartite graph 70B using a processing procedure similar to that of the above embodiment.

[0127] That is, in step S101, the control unit 11 acquires multiple training graphs 30B. Each training graph 30B is composed of an undirected bipartite graph 70B. The elements (vertices) and edges in each training graph 30B may be set appropriately to appropriately represent the conditions of the training target. In step S102, the control unit 11 performs machine learning on the inference model 5 using the acquired multiple training graphs 30B. This machine learning can generate a trained inference model 5 that has acquired the ability to infer a solution to a task for the undirected bipartite graph 70B. The task may be, for example, the above-mentioned matching task or an inference task for an image. In step S103, the control unit 11 generates learning result data 125 indicating the generated trained inference model 5 and stores the generated learning result data 125 in a predetermined storage area. The learning result data 125 may be provided to the inference device 2B at any time.

[0128] (Inference device) The hardware and software configurations of inference device 2B may be similar to those of inference device 2 according to the above embodiment. In the second specific example, inference device 2B can infer a solution to a task for undirected bipartite graph 70B using a processing procedure similar to that of inference device 2 described above.

[0129] That is, in step S201, the control unit of the inference device 2B operates as an acquisition unit. An elephant graph 221B is obtained. The target graph 221B is composed of an undirected bipartite graph 70B. The elements (vertices) and edges in the target graph 221B may be set appropriately to appropriately express the conditions of the inference target. If the trained inference model 5 has acquired the ability to perform a predetermined inference on the above image, the control unit may obtain the target graph 221B (feature map) by performing calculations on the convolutional neural network.

[0130] In step S202, the control unit operates as an inference unit and infers a solution to the task for the acquired object graph 221B using the trained inference model 5. If the trained inference model 5 has acquired the ability to perform the matching task, the control unit can obtain the inference result of the matching. If the trained inference model 5 has acquired the ability to perform a predetermined inference for the image, the control unit can obtain the result of the predetermined inference for the image from which the object graph 221B was acquired.

[0131] In step S203, the control unit operates as an output unit and outputs information related to the inference result. As an example, the control unit may output the obtained inference result directly to an output device. As another example, when a matching inference result is obtained, the control unit may establish matching for at least a portion of the inference result, as in the first specific example. In this case, as in the first specific example, the inference device 2B may be configured to perform matching online in real time. For example, in the case of the automatic check-in / seat allocation, the inference device 2B may add corresponding elements to each subset as guests arrive and rooms / seats become available, and assign vacant rooms / seats through matching in response to the guests' requests. Then, the inference device 2B may delete each element corresponding to the assigned guest and room / seat from each subset. As yet another example, in the case of object tracking, the control unit may track one or more objects in each image based on the inference result. In this case, inference device 2B may be configured to add a target object to the matching targets when it is detected in the image, and to remove the target object from the matching targets when it is no longer detected in the image (e.g., when it moves out of the shooting range or is hidden behind an obstruction, etc.), thereby enabling inference device 2B to perform object tracking in real time.

[0132] (Features) According to the second specific example, inference processing can be made more efficient in a situation where a given task (e.g., a matching task, an inference task for an image) is represented by an undirected bipartite graph 70B. The model generation device 1 can generate a trained inference model 5 that can efficiently perform a task represented by an undirected bipartite graph 70B. By using such a trained inference model 5, the inference device 2B can efficiently infer a solution to the task represented by the undirected bipartite graph 70B.

[0133] Also, as in the first specific example, in the undirected bipartite graph 70B, edges do not necessarily need to be assigned to all combinations of elements between the two subsets (71B, 72B). The selection module 503 may select one or more edges extending from each element, regardless of whether edges have been assigned in advance between the elements. This makes it possible to refer to combinations of elements that are useful for solving a task and infer a solution to the task, even if edges have not been assigned. As a result, improved inference accuracy can be expected.

[0134] (C) When using a general directed graph FIG. 12 schematically illustrates an example of an application scenario of an inference system 100C according to a third specific example. The third specific example is an example in which the above-described embodiment is applied to a scenario in which a general directed graph (which may simply be referred to as a directed graph) is used as a graph representing a given condition (a target for solving a task). The inference system 100C according to the third specific example includes a model generation device 1 and an inference device 2C. The inference device 2C is an example of the inference device 2 described above.

[0135] In the third specific example, each training graph 30C and target graph 221C (input graph) is a general directed graph 70C. The general directed graph 70C may be composed of multiple elements (vertices) and one or more directed edges connecting the elements (vertices). Circles, squares, and stars in the figure indicate elements (vertices). The multiple sets in the input graph are composed of a first set and a second set. The elements belonging to the first set correspond to the starting points of the directed edges constituting the general directed graph 70C, and the elements belonging to the second set correspond to the ending points of the directed edges. Therefore, the number of elements in the first set and the second set may be the same, and the elements belonging to each set may be the same. An edge flowing from an element belonging to the first set to an element belonging to the second set corresponds to a directed edge flowing from the starting point to the ending point, and an edge flowing from an element belonging to the second set to an element belonging to the first set corresponds to a directed edge flowing from the starting point to the ending point. The training graph 30C and the target graph 221C are examples of the training graph 30 and the target graph 221 in the above embodiment. The number of elements in the general directed graph 70C and the presence or absence of directed edges between the elements may be determined as appropriate to appropriately represent the target of solving the task.

[0136] As long as the task can be represented by a general directed graph, its content is not particularly limited and may be appropriately selected depending on the embodiment. In one example, the general directed graph 70C may be configured to represent a network. The network may correspond to a transportation network such as a road network, a rail network, a sea route network, or an air route network. Alternatively, the network may correspond to a communication network. Correspondingly, the task may be to infer (including predict) an event occurring on the network. Inferring an event may be, for example, predicting the time required to travel between two vertices, predicting the occurrence of an anomaly, searching for an optimal route, etc.

[0137] In this case, the characteristics of the directed edge may be configured to include attributes of the connection between corresponding vertices in the network. If the network corresponds to a transportation network, each vertex may correspond to a transportation hub (e.g., an intersection, a major location, a station, a port, an airport, etc.), and the attributes of the connection may be, for example, the type of road (e.g., a road type such as an ordinary road or an expressway), the road's traffic capacity, the time required for travel, the distance, etc. If the network corresponds to a communication network, each vertex may correspond to a communication hub (e.g., a server, a router, a switch, etc.), and the attributes of the connection may be, for example, the type of communication, the communication capacity, the communication speed, etc. The characteristics of the directed edge may be configured by one or more attribute values. Note that, in correspondence with predicting an event, the inference device 2C may be interpreted as a prediction device.

[0138] In another example, the task may be to divide a general directed graph (i.e., to classify the elements that make up the general directed graph). As a specific example of the task, the general directed graph 70C may be a weighted graph that indicates the similarity of motion obtained from the result of matching feature points between two consecutive images. In this case, the division of the graph may be performed to detect a dense subgraph for the purpose of determining the accuracy of the matching. Alternatively, the general directed graph 70C may be a graph that indicates feature points in an image. In this case, the division of the graph may be performed to classify the feature points by object. In each case, the features of the directed edges may be set appropriately to indicate the relationship between the feature points.

[0139] In yet another example, the task may be to infer features that appear in a general directed graph. As a specific example of the task, the general directed graph 70C may be a graph that represents a profile of an object (e.g., a person, an object, etc.), such as a knowledge graph. In this case, the task may be to infer features of the object that are derived from the profile represented in the graph.

[0140] The graph inference module 55 performs tasks (e.g., The training graph 30C may be configured as appropriate to derive inference results for solutions (e.g., event inference, graph partitioning, feature inference, etc.) from the graph information of each set. Correct answers in machine learning may be provided as appropriate depending on the content of the task for the general directed graph 70C. For example, correct answers for inference such as true values ​​of events, true values ​​of partitioning, and true values ​​of features may be provided as appropriate for each training graph 30C.

[0141] In the third specific example, during machine learning by the model generation device 1 and during inference processing by the inference device 2C, a general directed graph 70C is treated as an input graph. Calculating the feature values ​​of each element belonging to each set involves calculating the feature values ​​of the elements belonging to the first set from the features of the directed edges emanating from the elements belonging to the first set, and calculating the feature values ​​of the elements belonging to the second set from the features of the directed edges emanating from the elements belonging to the second set. That is, the feature extractor 501 is configured to calculate the feature values ​​of each element belonging to the first set from the features of the directed edges emanating from each element belonging to the first set, and to calculate the feature values ​​of each element belonging to the second set from the features of the directed edges emanating from each element belonging to the second set. Except for these points, the configuration of the third specific example may be the same as that of the above-described embodiment.

[0142] (Model generation device) In the third specific example, the model generation device 1 can generate a trained inference model 5 that has acquired the ability to infer solutions to tasks for the general directed graph 70C using a processing procedure similar to that of the above embodiment.

[0143] That is, in step S101, the control unit 11 acquires multiple training graphs 30C. Each training graph 30C is composed of a general directed graph 70C. The elements (vertices) and directed edges in each training graph 30C may be set appropriately to appropriately represent the conditions of the training target. In step S102, the control unit 11 performs machine learning on the inference model 5 using the acquired multiple training graphs 30C. This machine learning can generate a trained inference model 5 that has acquired the ability to infer a solution to a task for the general directed graph 70C. The task may be, for example, the above-mentioned event inference, graph partitioning, feature inference, etc. In step S103, the control unit 11 generates learning result data 125 indicating the generated trained inference model 5 and stores the generated learning result data 125 in a predetermined storage area. The learning result data 125 may be provided to the inference device 2C at any time.

[0144] (Inference device) The hardware and software configurations of the inference device 2C may be similar to those of the inference device 2 according to the above embodiment. In the third specific example, the inference device 2C can infer a solution to a task for the general directed graph 70C using the same processing procedure as that of the inference device 2 described above.

[0145] That is, in step S201, the control unit of the inference device 2C operates as an acquisition unit and acquires a target graph 221C. The target graph 221C is composed of a general directed graph 70C. The elements (vertices) and directed edges in the target graph 221C may be set appropriately to appropriately express the conditions of the inference target. In step S202, the control unit operates as an inference unit and uses a trained inference model 5 to infer a solution to the task for the acquired target graph 221C.

[0146] In step S203, the control unit operates as an output unit and outputs information related to the inference result. As one example, the control unit may output the obtained inference result directly to an output device. As another example, when an inference result of an event occurring on the network is obtained, the control unit may output an alarm notifying the user if the possibility of an abnormality occurring is equal to or greater than a threshold. As yet another example, when an inference result of an optimal route in a communication network is obtained, the control unit may transmit a packet according to the inferred optimal route.

[0147] (Features) According to the third specific example, in a situation where a given task is represented by a general digraph 70C, it is possible to improve the efficiency of inference processing. The model generation device 1 can generate a trained inference model 5 that can efficiently perform the task represented by the general digraph 70C. By using such a trained inference model 5, the inference device 2C can efficiently infer a solution to the task represented by the general digraph 70C.

[0148] Also, as in the first specific example, in the general directed graph 70C, directed edges do not necessarily need to be assigned to all combinations of elements between the first set and the second set. The selection module 503 may select one or more edges extending from each element, regardless of whether edges have been assigned in advance between the elements. This allows for the solution of a task to be inferred by referring to combinations of elements that are useful for solving the task, even if edges have not been assigned. As a result, improved inference accuracy can be expected.

[0149] (D) When using a general undirected graph FIG. 13 schematically illustrates an example of an application scenario of an inference system 100D according to a fourth specific example. The fourth specific example is an example in which the above-described embodiment is applied to a scenario in which a general undirected graph (which may simply be referred to as an undirected graph) is used as a graph representing a given condition (a target for solving a task). The inference system 100D according to the fourth specific example includes a model generation device 1 and an inference device 2D. The inference device 2D is an example of the inference device 2 described above.

[0150] In the fourth specific example, each training graph 30D and object graph 221D (input graph) are general undirected graphs 70D. The general undirected graph 70D may be composed of multiple elements (vertices) and one or more edges connecting the elements (vertices). The multiple sets in the input graph are composed of a first set and a second set. The elements belonging to the first set and the second set correspond to the elements constituting the general undirected graph 70D. Therefore, as in the third specific example, the number of elements in the first set and the second set may be the same, and the elements belonging to each set may be the same. An edge extending from an element belonging to the first set to an element belonging to the second set, and an edge extending from an element belonging to the second set to an element belonging to the first set, both correspond to an edge (undirected edge) connecting two corresponding elements. Each training graph 30D and object graph 221D are examples of each training graph 30 and object graph 221 in the above embodiment. The number of elements in general undirected graph 70D and the presence or absence of edges between elements may be determined appropriately so as to appropriately represent the target of solving the task.

[0151] As long as the task can be expressed by a general undirected graph, its content is not particularly limited and may be selected appropriately depending on the embodiment. The task in the fourth specific example may be the same as that in the third specific example, except that the edges have no direction. That is, in one example, the general undirected graph 70D may be configured to represent a network. Accordingly, the task may be to infer (including predict) events occurring on the network. In this case, the edge characteristics may be configured to include attributes of connections between corresponding vertices in the network. Note that, in correspondence with predicting events, the inference device 2D may be read as a prediction device. In another example, the task may be to partition the general undirected graph. In yet another example, the task may be to infer features appearing in the general undirected graph.

[0152] The graph inference module 55 may be configured as appropriate to derive an inference result for a solution to a task (e.g., the above-mentioned event inference, graph partitioning, feature inference, etc.) for the general undirected graph 70D from the graph information of each set. A correct answer in machine learning may be provided as appropriate depending on the content of the task for the general undirected graph 70D. For example, a correct answer for inference, such as a true value of an event, a true value of partitioning, or a true value of a feature, may be provided as appropriate for each training graph 30C.

[0153] In the fourth specific example, during machine learning by the model generation device 1 and during inference processing by the inference device 2D, a general undirected graph 70D is treated as an input graph. Calculating the feature values ​​of each element belonging to each set is performed by calculating the feature values ​​of each element from the features of the edges connecting to each element. That is, the feature extractor 501 is configured to calculate the feature values ​​of each element for each set from the features of the edges connecting to each element. Note that in the fourth specific example, the calculation results for the first set and the second set may be the same. Therefore, the calculation of the feature values ​​by the feature extractor 501, the selection of edges by the selection module 503, and the inference processing by the graph inference module 55 may be performed commonly for the first set and the second set. That is, either one of the series of calculation processes for the first set or the series of calculation processes for the second set may be omitted, and the execution of either one of the calculation processes may be treated as having executed both calculation processes. Except for these points, the configuration of the fourth specific example may be the same as that of the above-described embodiment.

[0154] (Model generation device) In the fourth specific example, the model generation device 1 can generate a trained inference model 5 that has acquired the ability to infer solutions to tasks for the general undirected graph 70D using a processing procedure similar to that of the above embodiment.

[0155] That is, in step S101, the control unit 11 acquires multiple training graphs 30D. Each training graph 30D is composed of a general undirected graph 70D. The elements (vertices) and edges in each training graph 30D may be set appropriately to appropriately represent the conditions of the training target. In step S102, the control unit 11 performs machine learning of the inference model 5 using the acquired multiple training graphs 30D. This machine learning makes it possible to generate a trained inference model 5 that has acquired the ability to infer a solution to a task for the general undirected graph 70D. In step S103, the control unit 11 generates learning result data 125 indicating the generated trained inference model 5 and stores the generated learning result data 125 in a predetermined storage area. The learning result data 125 may be provided to the inference device 2D at any time.

[0156] (Inference device) The hardware and software configurations of inference device 2D may be similar to those of inference device 2 according to the above embodiment. In the fourth specific example, inference device 2D can infer a solution to a task for general undirected graph 70D using a processing procedure similar to that of inference device 2 described above.

[0157] That is, in step S201, the control unit of the inference device 2D operates as an acquisition unit and acquires a target graph 221D. The target graph 221D is composed of a general undirected graph 70D. The elements (vertices) and edges in the target graph 221D may be set appropriately to appropriately express the conditions of the inference target. In step S202, the control unit operates as an inference unit and uses a trained inference model 5 to infer a solution to the task for the acquired target graph 221D.

[0158] In step S203, the control unit operates as an output unit and outputs information related to the inference result. As one example, the control unit may output the obtained inference result directly to an output device. As another example, when an inference result of an event occurring on the network is obtained, the control unit may output an alarm notifying the user if the possibility of an abnormality occurring is equal to or greater than a threshold. As yet another example, when an inference result of an optimal route in a communication network is obtained, the control unit may transmit a packet according to the inferred optimal route.

[0159] (Features) According to the fourth specific example, in a situation where a given task is represented by a general undirected graph 70D, it is possible to improve the efficiency of inference processing. The model generation device 1 can generate a trained inference model 5 that can efficiently execute the task represented by the general undirected graph 70D. By using such a trained inference model 5, the inference device 2D can efficiently infer a solution to the task represented by the general undirected graph 70D.

[0160] Also, as in the first specific example, in the general undirected graph 70D, edges do not necessarily need to be assigned to all combinations of elements between the first set and the second set. The selection module 503 may select one or more edges extending from each element, regardless of whether edges have been assigned in advance between the elements. This allows for the solution of a task to be inferred by referring to combinations of elements that are useful for solving the task, even if edges have not been assigned. As a result, improved inference accuracy can be expected.

[0161] (E) When using a vertex feature graph where the relationships between elements are unknown 14 schematically illustrates an example of an application scenario of an inference system 100E according to a fifth specific example. The fifth specific example is an example in which the above-described embodiment is applied to a scenario in which a vertex feature graph is employed as a graph representing a given condition (a target for solving a task), and the relationships between the elements (vertices) constituting the employed vertex feature graph are unknown. The inference system 100E according to the fifth specific example includes a model generation device 1 and an inference device 2E. The inference device 2E is an example of the inference device 2 described above.

[0162] In the fifth specific example, each training graph 30E and the target graph 221E (input graph) is a vertex feature graph 70E including a plurality of elements (vertices), each element being configured to have an attribute (vertex feature). The type and number of attributes possessed by each element may be determined appropriately depending on the embodiment. Each element may have one or more types of attribute values. A weight is an example of an attribute, and a vertex-weighted graph is an example of a vertex feature graph 70E.

[0163] Each element (vertex) included in the vertex feature graph 70E may be assigned to each set as appropriate. The number of sets may be determined as appropriate depending on the embodiment. In one example, all elements included in the vertex feature graph 70E may be assigned to both the first set and the second set, and the elements belonging to the first set and the elements belonging to the second set may be the same. In another example, the elements belonging to the first set and the elements belonging to the second set may partially overlap. In yet another example, in the case of inferring correspondences between point cloud data, each set may correspond to the point cloud data, and the elements belonging to each set may correspond to each point constituting the point cloud data. In this case, the elements belonging to each set may not overlap. Each training graph 30E and the object graph 221E are examples of each training graph 30 and object graph 221 in the above embodiment. The number and attributes of elements (vertices) in the vertex feature graph 70E may be determined as appropriate to appropriately represent the target for solving the task.

[0164] The content of the task is not particularly limited as long as it can be expressed by a vertex feature graph in which the relationships between elements are unknown, and may be appropriately selected depending on the embodiment. In one example, the task may be to estimate the relationships between elements belonging to each set. Estimating the relationships may be, for example, determining whether points are the same in matching the point cloud data.

[0165] As a specific example of a task, the vertex feature graph 70E may be configured to correspond to an image in which objects have been detected (e.g., a scene graph). In this case, each element may correspond to a detected object, and the attributes of each element may correspond to the attributes of the detected object. The sets of the input graph may be configured by a first set and a second set, and the sets included in the vertex feature graph 70E may be All elements may be sorted into both the first set and the second set. Estimating the relationship between the elements may be inferring features that appear in the image (image recognition). This specific example may be employed in various situations where image recognition is performed.

[0166] As another specific example of the task, the vertex feature graph 70E may be configured to correspond to an image containing one or more people, in which feature points of the people have been detected (e.g., a scene graph or a skeletal model). In this case, each element may correspond to a feature point (e.g., a joint, etc.) of the detected person, and the attributes of each element may correspond to the attributes of the detected feature point (e.g., the type, position, tilt, etc.) of the joint. The multiple sets of the input graph may be composed of a first set and a second set, and all elements included in the vertex feature graph 70E may be assigned to both the first set and the second set. Estimating the relationship between the elements may be inferring the type of movement of a person (motion analysis). This specific example may be employed in various situations where motion analysis is performed, such as video analysis of sports (particularly team sports).

[0167] As another specific example of a task, the vertex feature graph 70E may be configured to indicate the properties of a composition. The properties of a composition may be defined, for example, by the type of effect on the human body (medicinal effect, side effects, residual effect, reaction upon skin contact, etc.), the degree of each effect (efficiency of photoelectric effect, efficiency of thermoelectric effect, infrared absorption / reflection efficiency, friction coefficient, etc.). In this case, each element may be a specific The elements may correspond to properties, and the attributes of each element may be configured to indicate the degree of a particular property. The sets of the input graph may be configured as a first set and a second set, and all elements included in the vertex feature graph 70E may be assigned to both the first set and the second set. Estimating the relationship between each element may involve inferring (generating) a structure (e.g., a chemical formula or a physical crystal structure) of a composition that satisfies the properties indicated by the vertex feature graph 70E.

[0168] As another specific example of the task, the vertex feature graph 70E may be configured with multiple pieces of point cloud data, each representing a two-dimensional or three-dimensional model. In this case, each set may correspond to a piece of point cloud data, and each element belonging to each set may correspond to a point included in the corresponding point cloud data. The attributes of each element may correspond to the attributes of the points included in the point cloud data. Estimating the relationship between each element may be, for example, matching points such as point cloud registration, optical flow estimation, or stereo matching.

[0169] The graph inference module 55 may be configured as appropriate to derive an inference result for a solution to a task (e.g., the above-mentioned image recognition, motion analysis, structure generation, matching, etc.) for the vertex feature graph 70E from the graph information of each set. A correct answer in machine learning may be provided as appropriate depending on the content of the task for the vertex feature graph 70E. For example, a correct answer for inference such as a true value for image recognition, a true value for motion analysis, or a true value for structure generation may be provided as appropriate for each training graph 30E.

[0170] In the fifth specific example, during machine learning by the model generation device 1 and during inference processing by the inference device 2E, the vertex feature graph 70E is treated as an input graph. Calculating the feature amount of each element belonging to each set is configured by calculating the feature amount of each element from the attributes of each element. That is, the feature extractor 501 is configured to calculate, for each set, the feature amount of each element from the attributes of each element. Except for these points, the configuration of the fifth specific example may be the same as that of the above-described embodiment.

[0171] (Model generation device) In the fifth specific example, the model generation device 1 can generate a trained inference model 5 that has acquired the ability to infer solutions to tasks for the vertex feature graph 70E using a processing procedure similar to that of the above embodiment.

[0172] That is, in step S101, the control unit 11 acquires a plurality of training graphs 30E. Each training graph 30E is composed of a vertex feature graph 70E. The elements (vertices) and attributes in each training graph 30E may be set appropriately to appropriately represent the conditions of the training target. In step S102, the control unit 11 performs machine learning of the inference model 5 using the acquired multiple training graphs 30E. This machine learning makes it possible to generate a trained inference model 5 that has acquired the ability to infer a solution to a task for the vertex feature graph 70E. In step S103, the control unit 11 generates learning result data 125 indicating the generated trained inference model 5 and stores the generated learning result data 125 in a predetermined storage area. The learning result data 125 may be provided to the inference device 2E at any time.

[0173] (Inference device) The hardware configuration and software configuration of the inference device 2E may be similar to those of the inference device 2 according to the above embodiment. In the fifth specific example, the inference device 2E can infer a solution to a task for the vertex feature graph 70E using the same processing procedure as that of the inference device 2 described above.

[0174] That is, in step S201, the control unit of the inference device 2E operates as an acquisition unit and acquires a target graph 221E. The target graph 221E is composed of a vertex feature graph 70E. The elements (vertices) and attributes in the target graph 221E may be set appropriately to appropriately express the conditions of the inference target. In the case of structure generation, the value of each property (i.e., the attribute value of each element) may be set as a random number or may be set by an operator. In step S202, the control unit operates as an inference unit and uses the trained inference model 5 to infer a solution to a task for the acquired target graph 221E. In one example, the task is to estimate the relationships between elements in the target graph 221E.

[0175] In step S203, the control unit operates as an output unit and outputs information related to the inference result. As an example, the control unit may output the obtained inference result directly to an output device. As another example, the control unit may perform information processing according to the obtained inference result. For example, if the object graph 221E is configured to correspond to an image in which an object has been detected, the control unit may perform information processing according to the result of image recognition. As a specific application scenario, the fifth specific example may be applied to a scenario in which a captured image obtained by capturing an image of the situation outside the vehicle with a camera is recognized. In this scenario, if it is determined based on the result of image recognition that an obstacle is approaching the vehicle, the control unit may issue a command to the vehicle to avoid the obstacle (e.g., to stop, change lanes, etc.). Furthermore, for example, if the object graph 221E is configured to correspond to an image in which feature points of a person are detected, the control unit may perform information processing according to the result of motion analysis. As a specific application scenario, the fifth specific example may be applied to a scenario in which motion analysis of a person appearing in a captured image obtained by a camera installed on a platform at a train station is performed. In this situation, if it is determined based on the results of the motion analysis that a person on the platform is in imminent danger (for example, is about to fall off the platform), the control unit may output an alarm to notify that fact.

[0176] (Features) According to the fifth specific example, in a situation where a given task (e.g., estimating relationships between elements) is represented by a vertex feature graph 70E, it is possible to improve the efficiency of inference processing. The model generation device 1 can generate a trained inference model 5 that can efficiently perform the task represented by the vertex feature graph 70E. By using such a trained inference model 5, the inference device 2D can efficiently infer a solution to the task represented by the vertex feature graph 70E.

[0177] (F) When using a vertex feature graph in which the relationships between elements are defined FIG. 15 is a diagram illustrating an example of an application scene of the inference system 100F according to the sixth example. The sixth specific example is an example in which the above-described embodiment is applied to a situation in which a vertex feature graph is used as a graph representing a given condition (a target for solving a task), and the relationships between elements (vertices) included in the vertex feature graph used are known. An inference system 100F according to the sixth specific example includes a model generation device 1 and an inference device 2F. The inference device 2F is an example of the inference device 2 described above.

[0178] In the sixth specific example, each training graph 30F and object graph 221F (input graph) is a vertex feature graph 70F including a plurality of elements (vertices), each element having an attribute. In the sixth specific example, the relationships between the elements included in the vertex feature graph 70F are defined. Except for the fact that the relationships are defined in advance, the vertex feature graph 70F may be configured similarly to the vertex feature graph 70E in the fifth specific example. Each training graph 30F and object graph 221F are examples of each training graph 30 and object graph 221 in the above embodiment. Note that the number of elements (vertices), attributes, and edges between elements (vertices) in the vertex feature graph 70F may be determined appropriately to appropriately represent the target of solving the task.

[0179] As long as the task can be expressed by a vertex feature graph, the content of the task is not particularly limited and may be appropriately selected depending on the embodiment. In one example, the task may be to infer an event derived from the vertex feature graph 70F.

[0180] As a specific example of a task, the vertex feature graph 70F may be configured to represent a chemical formula or a physical crystal structure indicating a composition. In this case, each element corresponds to an element, and the attributes of each element may correspond to element attributes, such as the type of element. The relationships between elements may correspond to the connection relationships between elements. The multiple sets of the input graph may be composed of a first set and a second set, and all elements included in the vertex feature graph 70F may be assigned to both the first set and the second set. Inferring an event may involve inferring characteristics related to the composition, such as the properties of the composition. Alternatively, inferring an event may involve inferring a procedure for producing a substance with that composition. For example, natural language, a symbol string, or the like may be used to express the procedure. When a symbol string is used, the correspondence between the symbol string and the procedure may be predetermined by a rule. Furthermore, the symbol string may include instructions for controlling the operation of a production device configured to produce a substance with that composition. The production device may be, for example, a computer, a controller, a robotic device, or the like. The instructions may be, for example, control instructions, computer programs, etc.

[0181] As another specific example of the task, the vertex feature graph 70F may be configured to indicate the characteristics of each of multiple objects (e.g., people, objects, etc.). In this case, each element may correspond to each object, and the attributes of each element may correspond to the characteristics of each object. The characteristics of each object may be extracted from a graph representing a profile using the third or fourth specific example. The multiple sets of the input graph may be composed of a first set and a second set, and all elements included in the vertex feature graph 70F may be assigned to both the first and second sets. Inferring an event may be optimally matching objects with each other. The objects to be matched may be, for example, a customer / advertisement, a patient / doctor, a person / job, etc.

[0182] As yet another specific example of the task, the vertex feature graph 70F may be configured to represent an electric power supply network. In this case, each element may correspond to an electric power production node or a consumption node. The attributes of the element may be configured to represent, for example, the production amount, the consumption amount, etc. Each edge may correspond to an electric wire. The multiple sets of the input graph may be composed of a first set and a second set, and all elements included in the vertex feature graph 70F may be assigned to both the first set and the second set. Inferring an event may be inferring an optimal combination of production nodes and consumption nodes.

[0183] The graph inference module 55 may be configured as appropriate to derive an inference result for a solution to a task (e.g., the above-mentioned event inference) for the vertex feature graph 70F from the graph information of each set. A correct answer in machine learning may be provided as appropriate depending on the content of the task for the vertex feature graph 70F. For example, a correct answer for inference, such as a true value of an event, may be provided for each training graph 30F.

[0184] In the sixth specific example, during machine learning by the model generation device 1 and during inference processing by the inference device 2F, the vertex feature graph 70F is treated as an input graph. Calculating the feature amount of each element belonging to each set is configured by calculating the feature amount of each element from the attributes of each element and information indicating the relationships between the edges connecting to each element. That is, the feature extractor 501 is configured to calculate the feature amount of each element for each set from the attributes of each element and the edge features (information indicating the relationships). Except for these points, the configuration of the sixth specific example may be the same as that of the above-described embodiment.

[0185] (Model generation device) In the sixth specific example, the model generation device 1 can generate a trained inference model 5 that has acquired the ability to infer solutions to tasks for the vertex feature graph 70F using a processing procedure similar to that of the above embodiment.

[0186] That is, in step S101, the control unit 11 acquires multiple training graphs 30F. Each training graph 30F is composed of a vertex feature graph 70F. The elements (vertices), attributes, and edges in each training graph 30F may be set appropriately to appropriately represent the conditions of the training target. In step S102, the control unit 11 performs machine learning of the inference model 5 using the acquired multiple training graphs 30F. This machine learning can generate a trained inference model 5 that has acquired the ability to infer a solution to a task for the vertex feature graph 70F. In step S103, the control unit 11 generates learning result data 125 indicating the generated trained inference model 5 and stores the generated learning result data 125 in a predetermined storage area. The learning result data 125 may be provided to the inference device 2F at any timing.

[0187] (Inference device) The hardware configuration and software configuration of the inference device 2F may be the same as those of the inference device 2 according to the above embodiment. In the sixth specific example, the inference device 2F can infer a solution to a task for the vertex feature graph 70F using the same processing procedure as that of the inference device 2 described above.

[0188] That is, in step S201, the control unit of the inference device 2F operates as an acquisition unit and acquires a target graph 221F. The target graph 221F is composed of a vertex feature graph 70F. The elements (vertices), attributes, and edges in the target graph 221F may be set appropriately to appropriately express the conditions of the inference target. In step S202, the control unit operates as an inference unit and uses the trained inference model 5 to infer a solution to the task for the acquired target graph 221F.

[0189] In step S203, the control unit operates as an output unit and outputs information related to the inference result. As one example, the control unit may output the obtained inference result directly to an output device. As another example, if the object graph 221F is configured to indicate the composition (chemical formula or physical crystal structure) of the object and a procedure for producing a substance with the object composition is inferred, the control unit may control the operation of a production device according to the procedure indicated by the obtained inference result. As another example, if a matching between objects or a combination of a production node and a consumption node is inferred, the control unit may establish (confirm) the matching or combination.

[0190] (Features) According to the sixth concrete example, in a situation where a given task is represented by a vertex feature graph 70F, it is possible to improve the efficiency of inference processing. The model generation device 1 can generate a trained inference model 5 that can efficiently execute the task represented by the vertex feature graph 70F. By using such a trained inference model 5, the inference device 2F can efficiently infer a solution to the task represented by the vertex feature graph 70F.

[0191] Also, as in the first specific example, in the vertex feature graph 70F, edges do not necessarily have to be assigned to all combinations of elements between sets. The selection module 503 may select one or more edges extending from each element, regardless of whether edges have been assigned in advance between elements. This makes it possible to refer to combinations of elements that are useful for solving a task and infer a solution to the task, even if edges have not been assigned. As a result, improvement in inference accuracy can be expected.

[0192] (G) When hypergraphs are used FIG. 16 shows a schematic example of an application scenario of an inference system 100G according to a seventh specific example. The seventh specific example is an example in which the above-described embodiment is applied to a scenario in which a hypergraph is used as a graph representing a given condition (a target for solving a task). The inference system 100G according to the seventh specific example includes a model generation device 1 and an inference device 2G. The inference device 2G is an example of the inference device 2.

[0193] In the seventh specific example, each training graph 30G and object graph 221G (input graph) is a K-part hypergraph 70G. K may be an integer greater than or equal to 3. Multiple sets of input graphs are composed of K subsets of the hypergraph 70G. Multiple elements (vertices) included in the hypergraph 70G may be divided to belong to one of the K subsets. In the example of FIG. 16, each element included in the hypergraph 70G is divided to belong to one of three subsets. Branches may be appropriately provided to indicate combinations of elements belonging to each subset. Each training graph 30G and object graph 221G are examples of each training graph 30 and object graph 221 in the above embodiment. Note that the number of elements (vertices) in the hypergraph 70G, the presence or absence of branches between elements (vertices), and the value of K may be appropriately determined to appropriately represent the target for solving the task.

[0194] As long as the task can be represented by a hypergraph, its content may not be particularly limited and may be selected appropriately depending on the embodiment. For example, the task may be a matching task between K parties, in which the optimal combination of objects belonging to each party is determined. In this case, each subset of the hypergraph 70G corresponds to each party in the matching task. Elements belonging to each subset of the hypergraph 70G correspond to objects belonging to each party. The characteristics of the branches flowing from the i-th element belonging to the i-th subset to combinations of elements belonging to each of the (K-1) other subsets (other than the i-th) may correspond to the cost or reward of matching the object corresponding to the i-th element and the combination of objects corresponding to each element belonging to the other subsets other than the i-th. Alternatively, the characteristics of the branches may correspond to the desirability of the object corresponding to the i-th element to the combination of objects corresponding to each element belonging to the other subsets other than the i-th.

[0195] The matching target may be selected as appropriate depending on the embodiment. The matching task may be, for example, identifying an optimal combination of three or more parts, identifying an optimal combination of three or more people, identifying an optimal triplet combination for learning to calculate triplet loss in metric learning, identifying a combination of feature points belonging to the same person when simultaneously inferring the poses of multiple people in an image, identifying an optimal combination of delivery trucks, packages, and routes, etc. The matching may be performed for the purpose of identifying optimal combinations, etc. Accordingly, the target of matching may be, for example, a part, a person, data for which triplet loss is to be calculated, feature points of a person detected in an image, a delivery truck / package / route, etc. Identifying the optimal combination of people may be performed for team building, for example, between a manager / programmer / salesperson, a web front-end programmer / back-end programmer / data scientist, etc.

[0196] The graph inference module 55 may be configured to derive an inference result for a solution to a task (e.g., the above-mentioned matching task) for the hypergraph 70G from the graph information of each subset. A correct answer in machine learning may be assigned as appropriate depending on the content of the task for the hypergraph 70G. For example, a correct answer for inference, such as a true value of matching that satisfies a predetermined criterion, may be assigned to each training graph 30G. As an example, in the case of the above-mentioned team building, the predetermined criterion may be defined to evaluate results achieved in the real world, such as financial benefits, reduced employee turnover, reduced development time, increased number of progress reports, and improvements in psychological indicators (e.g., questionnaires, sleep time, blood pressure, etc.).

[0197] In the seventh specific example, during machine learning by the model generation device 1 and during inference processing by the inference device 2G, the hypergraph 70G is treated as an input graph. Calculating the feature amount of each element belonging to each set is performed by calculating the feature amount of each element from the features of the edges connecting to each element belonging to each subset of the hypergraph 70G. That is, the feature extractor 501 is configured to calculate the feature amount of each element for each subset of the hypergraph 70G from the features of the edges connecting to each element. Except for these points, the configuration of the seventh specific example may be the same as that of the above-described embodiment.

[0198] (Model generation device) In the seventh specific example, the model generation device 1 can generate a trained inference model 5 that has acquired the ability to infer solutions to tasks for the hypergraph 70G using a processing procedure similar to that of the above embodiment.

[0199] That is, in step S101, the control unit 11 acquires multiple training graphs 30G. Each training graph 30G is composed of a hypergraph 70G. The element and edge settings in each training graph 30G may be appropriately set to appropriately represent the conditions of the training target. In step S102, the control unit 11 performs machine learning of the inference model 5 using the acquired multiple training graphs 30G. This machine learning makes it possible to generate a trained inference model 5 that has acquired the ability to infer a solution to a task for the hypergraph 70G. In step S103, the control unit 11 generates learning result data 125 indicating the generated trained inference model 5 and stores the generated learning result data 125 in a predetermined storage area. The learning result data 125 may be provided to the inference device 2G at any time.

[0200] (Inference device) The hardware and software configurations of the inference device 2G may be similar to those of the inference device 2 according to the above embodiment. In the seventh specific example, the inference device 2G can infer a solution to a task for the hypergraph 70G using a processing procedure similar to that of the inference device 2 described above.

[0201] That is, in step S201, the control unit of the inference device 2G operates as an acquisition unit and acquires a target graph 221G. The target graph 221G is composed of a hypergraph 70G. The elements and edges in the target graph 221G may be set appropriately to appropriately express the conditions of the inference target. In step S202, the control unit operates as an inference unit and uses the trained inference model 5 to infer a solution to the task for the acquired target graph 221G.

[0202] In step S203, the control unit operates as an output unit and outputs information related to the inference result. As one example, the control unit may output the obtained inference result directly to an output device. As another example, when the inference result of the above matching is obtained, the control unit may establish matching for at least a portion of the inference result, as in the first specific example, etc. Furthermore, the inference device 2G may be configured to perform matching online in real time.

[0203] (Features) According to the seventh concrete example, in a situation where a given task is represented by a hypergraph 70G, the efficiency of the inference process can be improved. The model generation device 1 can generate a trained inference model 5 that can efficiently execute the task represented by the hypergraph 70G. The inference device 2G can use such a trained inference model 5 to efficiently infer a solution to the task represented by the hypergraph 70G.

[0204] Also, as in the first specific example, in the hypergraph 70G, edges do not necessarily need to be assigned to all combinations of elements between subsets. The selection module 503 may select one or more edges connecting to each element, regardless of whether edges have been assigned in advance between elements. This allows for the solution of a task to be inferred by referring to combinations of elements that are useful for solving the task, even if edges have not been assigned. As a result, improved inference accuracy can be expected.

[0205] <4.2> In the above embodiment, the selection module 503 may include a differentiable calculation module configured to calculate the likelihood. That is, the selection module 503 may be configured to calculate the likelihood between each element and each candidate element using the calculation module. The calculation module may be configured as a machine learning model including one or more parameters for performing a calculation process to calculate the likelihood, the parameters being adjusted by machine learning. In the above machine learning process, the calculation module of the selection module 503 may also be trained consistently with the graph inference module 55 and the feature extractor 501. This allows the performance of the calculation module to be optimized for the inference process. As a result, the trained inference model 5 can ensure the accuracy of inference while improving the efficiency of the inference process.

[0206] <4.3> In the above embodiment, during the forward calculation of machine learning in step S102, the control unit 11 may add a random number to the likelihood calculated in the selection module 503 and select one or more branches based on the obtained value. This allows for a probabilistic search for branch selection in the machine learning process. In other words, it is possible to expand the range for searching for optimal values ​​for the branch selection parameters. As a result, it is possible to expect an improvement in the inference accuracy of the generated trained inference model 5.

[0207] <4.4> In the above embodiment, the selection module 503 may have multiple criteria for selecting branches, and may be configured to select branches for each criterion. The multiple criteria may include, for example, prioritizing branches with high likelihood, selecting branches whose likelihoods differ by more than a threshold, etc. One of the multiple criteria may include random selection. In this case, the selection module 503 may be configured to further add information indicating the criteria for selecting a branch to information indicating the branch selection result in the graph information of each set (for example, by performing a cat operation). The information indicating the criteria may be configured by, for example, a one-hot vector. Each criterion may be expressed by an arbitrary function. Two or more criteria may be expressed by one function, or one criterion may be expressed by one function. may be expressed as a single function. According to this modification, by selecting branches based on multiple criteria, the range of branches to be referenced can be expanded, which is expected to improve the inference accuracy of the trained inference model 5 that is generated.

[0208] §5 Working Examples In order to verify the effectiveness of the above embodiment, inference models according to the following examples and comparative examples were generated, although the present invention is not limited to the following examples.

[0209] <Dataset> First, we assumed the vertex feature graph from the fifth example above as a graph representing the given conditions, and adopted matching between two point cloud data as the task. For machine learning and evaluation of the matching task, we prepared the following two datasets.

[0210] The first dataset includes reference 2 (Gul Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J. Black, Ivan Laptev, Cordelia Schmid, "Learning from Synthetic Humans", [online], [Retrieved January 11, 2022]), and the Internet.<URL: https: / / arxiv.org / abs / 1701.01370> We used the Surreal dataset published by the National Institute of Standards and Technology (NIS), which is part of the National Institute of Standards and Technology (NIST). This first dataset contains 230,000 point cloud samples for training and 100 point cloud samples for evaluation. 115,000 combinations (training graph) were created by randomly pairing the training point cloud samples. 100 combinations (input graph and target graph for evaluation) were created by randomly rotating and translating the evaluation point cloud samples to create paired point cloud samples. As a result, each point cloud sample was configured to represent an object (human body) with no change in shape between each combination (rigid body setting).

[0211] The second dataset includes the Surreal dataset from the first dataset mentioned above and reference 3. (Bo Li, et al. "A comparison of 3D shape retrieval methods based on a large-scale benchmark supporting multimodal queries", [online], [January 11, 2020) Search, Internet<URL: https: / / userweb.cs.txstate.edu / ~burtscher / papers / cviu15.pdf> ) was used. This second dataset includes: The second dataset contains 230,000 and 860 point cloud samples, respectively. The same 115,000 combinations of point cloud samples were used for training as in the first dataset. Meanwhile, 430 combinations (input graph and target graph for evaluation) of point cloud samples were created by randomly pairing them for evaluation. As a result, between each evaluation combination in the second dataset, each point cloud sample was configured to represent a human body with varying shape (posture) (non-rigid body setting). Note that in both the first and second datasets, each point cloud sample contained 1,024 points (elements).

[0212] <Example> An inference model according to the first embodiment was created on a personal computer using an open-source machine learning library and employing the configuration according to the above embodiment. The feature extractor was configured to calculate 128-dimensional features for each element. The selection module was configured to select 128 edges for each element of each point cloud sample (set) based on likelihood. The likelihood was calculated using the above formula (1). The graph inference module employed WeaveNet, as proposed in Non-Patent Document 3. The number of feature weaving layers was set to 10. Machine learning for the inference model according to the first embodiment was performed using the above combinations of 115,000 training point cloud samples. In the machine learning, the learning rate was set to 0.0001, and Adam was used as the optimization algorithm. Then, in accordance with the method shown in Figure 4 of Reference 4 described below, the coordinates of each point were mapped to the corresponding destination based on the inference results for the combinations of training point cloud samples, and then optimization was performed to minimize the reconstruction error when the original point cloud was restored by a separate process. The parameter values ​​of the inference model were optimized (unsupervised learning), thereby generating a trained inference model according to the first embodiment.

[0213] Additionally, an inference model according to a second embodiment, having the same configuration as that of the first embodiment, was created. Then, correct labels were assigned to the combinations of the 115,000 training point cloud samples, and the parameter values ​​of the inference model according to the second embodiment were optimized (supervised learning) so as to obtain an output that matches the true value of matching indicated by the correct labels. The other conditions of the second embodiment were set to the same as those of the first embodiment. This resulted in the generation of a trained inference model according to the second embodiment.

[0214] <Comparative Example> On the other hand, Reference 4 (Yiming Zeng, Yue Qian, Zhiyu Zhu, Junhui Hou, Hui Yuan, Ying He, "CorrNet3D: Unsupervised end-to-end learning of dense correspondence for 3D point clouds", [online], [Retrieved January 11, 2022], Internet<URL: https: / / arxiv.org / abs / 2012.15638> ) was adopted to create an inference model according to the first comparative example. Then, machine learning was performed under the same conditions (unsupervised learning) as in the first embodiment, thereby generating a trained inference model according to the first comparative example. Also, an inference model according to a second comparative example, having a configuration similar to that of the first comparative example, was created. Then, machine learning was performed under the same conditions (supervised learning) as in the second embodiment, thereby generating a trained inference model according to the second comparative example.

[0215] <Evaluation> Using the trained inference models generated in the above manner for each Example and Comparative Example, a point cloud matching task was performed on a combination of evaluation point cloud samples from the first and second datasets. The tolerance was varied between 0.00 and 0.2 to evaluate the matching accuracy of the trained inference models for each Example and Comparative Example.

[0216] FIG. 17 shows experimental results (matching accuracy within a set tolerance) for each Example and Comparative Example for the first dataset. FIG. 18 shows experimental results (matching accuracy within a set tolerance) for each Example and Comparative Example for the second dataset. As shown in FIGS. 17 and 18, the Examples had higher matching accuracy than the Comparative Examples for both rigid and non-rigid settings. In particular, the matching accuracy of the first Example, which was trained by unsupervised learning, was higher than that of the second Comparative Example, which was trained by supervised learning. These results demonstrate that the present invention not only improves the efficiency of inference processing through branch selection, but also makes it possible to generate a trained inference model capable of inference with higher accuracy than existing methods.

[0217] Furthermore, to compare the matching accuracy between the first example and the first comparative example, two samples were extracted from the evaluation combinations of the second data set, in which the matching accuracy of the first example with an allowable error of 0.00 was comparable to the matching accuracy of the first comparative example with an allowable error of 0.06. Then, for each of the extracted combination samples, the matching accuracy of the first example and the first comparative example was compared when the allowable error was 0.00 and when the allowable error was 0.06.

[0218] 19A and 19C show the matching results of the first comparative example for the first combined sample of the second data set when the tolerance is set to 0.00 (FIG. 19A) and 0.06 (FIG. 19C). 19B and 19D show the matching results of the first embodiment for the first combined sample of the second data set when the tolerance is set to 0.00 (FIG. 19B) and 0.06 (FIG. 19D). 20A and 20C show the matching results of the first comparative example for the second combined sample of the second data set when the tolerance is set to 0.00 (FIG. 20A) and 0.06 (FIG. 20C). 19A to 19D and 20A to 20D show the matching results of the first embodiment for the second combined sample of the second data set when the tolerance is set to 0.00 (FIG. 20B) and 0.06 (FIG. 20D). Note that the solid lines in FIGS. 19A to 19D and 20A to 20D indicate pairs of points that are correctly matched within the set tolerance.

[0219] 19C, 19D, 20C, and 20D, for each sample in which the matching accuracy of the first embodiment with a tolerance of 0.00 was comparable to that of the first comparative example with a tolerance of 0.06, the matching accuracy of the first embodiment with a tolerance of 0.06 was even higher. These results also demonstrate that the present invention can generate a trained inference model that can perform inference with higher accuracy than existing methods. [Explanation of symbols]

[0220] 1...Model generation device, 11...control unit, 12...storage unit, 13...communication interface, 14...External interface, 15...input device, 16...output device, 17...drive, 81...model generation program, 91...storage medium, 111...acquisition unit, 112...learning processing unit, 113...storage processing unit, 125...Learning result data, 2... Reasoning device, 21...control unit, 22...storage unit, 23...communication interface, 24...External interface, 25...input device, 26...output device, 27...drive, 82...inference program, 92...storage medium, 211...acquisition unit, 212...inference unit, 213...output unit, 221...target graph, 30...Training graph, 5...inference model, 50...pre-processing module, 501... feature extractor, 503... selection module, 55...Graph inference module< / url:>

Claims

1. The computer obtaining a plurality of training graphs; performing machine learning of an inference model using the obtained training graphs; A model generation method that performs the inference model comprises a preprocessing module and a graph inference module; the pre-processing module comprises a feature extractor and a cull module; the feature extractor is configured to calculate a feature value of each element belonging to each of a plurality of sets included in the input graph; The selection module: Selecting one or more branches extending from each of the elements based on the calculated feature amount of each of the elements; and The method is configured to generate graph information indicating the calculated feature amount of each element and the selected one or more branches for each set, the graph inference module is configured to be differentiable and to infer a solution to a task for the input graph from the generated graph information of each of the sets; The machine learning is configured by inputting each of the training graphs as the input graph to the preprocessing module, and training the inference model so that an inference result obtained from the graph inference module matches a correct answer to the task for each of the training graphs. Model generation method.

2. Selecting the one or more branches based on the feature amount of each of the elements Calculating the likelihood between each element and each candidate element that is a candidate for connecting an edge from each element; and selecting the one or more branches according to the calculated likelihood; It is composed of The model generation method of claim 1 .

3. each training graph is a directed bipartite graph; A plurality of elements included in the directed bipartite graph are divided so as to belong to one of two subsets, the sets of the input graph are composed of the two subsets of the directed bipartite graph; calculating the feature amount of each element includes calculating the feature amount of each element from features of edges extending from each element of the directed bipartite graph as a starting point; The model generation method according to claim 1 or 2.

4. the task is a matching task between two parties, determining an optimal pairing between objects belonging to each party; the two subsets of the directed bipartite graph correspond to the two parties in the matching task; The elements belonging to each subset of the directed bipartite graph correspond to the objects belonging to each party. The model generation method according to claim 3 .

5. each training graph is an undirected bipartite graph; A plurality of elements included in the undirected bipartite graph belong to one of two subsets. It is divided into the sets of input graphs are composed of the two subsets of the undirected bipartite graph; calculating the feature amount of each element includes calculating the feature amount of each element from features of edges connected to each element of the undirected bipartite graph; The model generation method according to claim 1 or 2.

6. the task is a matching task between two parties, determining an optimal pairing between objects belonging to each party; the two subsets of the undirected bipartite graph correspond to the two parties in the matching task; The elements belonging to each subset of the undirected bipartite graph correspond to the objects belonging to each party. The model generation method according to claim 5 .

7. each training graph is a directed graph; the plurality of sets of the input graph are composed of a first set and a second set; the elements belonging to the first set correspond to the starting points of the directed edges constituting the directed graph; the elements belonging to the second set correspond to the end points of the directed branches; calculating the feature amounts of the elements includes calculating the feature amounts of the elements belonging to the first set from features of directed branches flowing out from the elements belonging to the first set, and calculating the feature amounts of the elements belonging to the second set from features of directed branches flowing into the elements belonging to the second set. The model generation method according to claim 1 or 2.

8. each training graph is an undirected graph; the plurality of sets of the input graph are composed of a first set and a second set; each element belonging to the first set and each element belonging to the second set corresponds to each element constituting the undirected graph; calculating the feature amount of each element includes calculating the feature amount of each element from features of branches connected to each element; The model generation method according to claim 1 or 2.

9. each training graph is configured such that each element included in each training graph has an attribute; calculating the feature amount of each element includes calculating the feature amount of each element from an attribute of each element; The model generation method according to claim 1 or 2.

10. the task is to estimate the relationships between elements belonging to each of the sets; The model generation method of claim 9.

11. each training graph is configured such that each element included in each training graph has an attribute; A relationship between each of the elements included in each of the training graphs is defined; calculating the feature amount of each element includes calculating the feature amount of each element from the attributes of each element and information indicating the relationship; The model generation method according to claim 1 or 2.

12. each said training graph is a hypergraph; A plurality of elements included in the hypergraph are divided so as to belong to any one of three or more subsets; the plurality of sets of the input graph are composed of the three or more subsets of the hypergraph; calculating the feature amount of each element includes calculating the feature amount of each element from features of edges connected to each element of the hypergraph; The model generation method according to claim 1 or 2.

13. an acquisition unit configured to acquire a plurality of training graphs; a learning processing unit configured to perform machine learning of an inference model using the acquired training graphs; A model generation device comprising: the inference model comprises a preprocessing module and a graph inference module; the pre-processing module comprises a feature extractor and a cull module; the feature extractor is configured to calculate a feature value of each element belonging to each of a plurality of sets included in the input graph; The selection module: Selecting one or more branches extending from each of the elements based on the calculated feature amount of each of the elements; and The method is configured to generate graph information indicating the calculated feature amount of each element and the selected one or more branches for each set, the graph inference module is configured to be differentiable and to infer a solution to a task for the input graph from the generated graph information of each of the sets; The machine learning is configured by inputting each of the training graphs as the input graph to the preprocessing module, and training the inference model so that an inference result obtained from the graph inference module matches a correct answer to the task for each of the training graphs. Model generation device.

14. On the computer, obtaining a target graph; Inferring a solution to a task for the obtained object graph using a machine learning trained inference model; outputting information about the results of inferring a solution to the task; An inference program for executing the inference model comprises a preprocessing module and a graph inference module; the pre-processing module comprises a feature extractor and a cull module; the feature extractor is configured to calculate a feature value of each element belonging to each of a plurality of sets included in the input graph; The selection module: Selecting one or more branches extending from each of the elements based on the calculated feature amount of each of the elements; and The method is configured to generate graph information indicating the calculated feature amount of each element and the selected one or more branches for each set, the graph inference module is configured to be differentiable and to infer a solution to the task for the input graph from the generated graph information of each of the sets; Inferring a solution to a task for the object graph comprises inputting the object graph as the input graph to the preprocessing module and obtaining a result of inferring a solution to the task from the graph inference module. Inference program.

15. an acquisition unit configured to acquire a target graph; an inference unit configured to infer a solution to a task for the obtained object graph using an inference model trained by machine learning; an output unit configured to output information regarding the result of inferring a solution to the task; An inference device comprising: the inference model comprises a preprocessing module and a graph inference module; the pre-processing module comprises a feature extractor and a cull module; the feature extractor is configured to calculate a feature value of each element belonging to each of a plurality of sets included in the input graph; The selection module: Selecting one or more branches extending from each of the elements based on the calculated feature amount of each of the elements; and The method is configured to generate graph information indicating the calculated feature amount of each element and the selected one or more branches for each set, the graph inference module is configured to be differentiable and to infer a solution to the task for the input graph from the generated graph information of each of the sets; Inferring a solution to a task for the object graph comprises inputting the object graph as the input graph to the preprocessing module and obtaining a result of inferring a solution to the task from the graph inference module. Reasoning device.

Images