Devices, data structure and computer-implemented methods for machine learning using semantic technologies
The method addresses the challenge of data-efficient labeling in machine learning by predicting tuples within data structures, assessing their uncertainty, and selectively labeling them, resulting in improved link prediction accuracy and resource efficiency.
Patent Information
- Application Number
- JP2024195894
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-11-10
- Filing Date
- 2024-11-08
- Publication Date
- 2025-05-22
AI Technical Summary
Current machine learning approaches face challenges in data-efficient labeling and training, particularly in predicting links within complex data structures like knowledge graphs, where providing accurate labels is costly and time-consuming.
The proposed method involves predicting tuples within a data structure using a trained model, assessing the uncertainty of each predicted tuple, and selectively labeling tuples based on their predicted uncertainty, thereby optimizing the labeling process and improving link prediction accuracy.
This approach enables data-efficient and resource-saving labeling by focusing on the most uncertain and informative tuples, leading to improved performance in link prediction and enabling self-learning systems to process heterogeneous data sources effectively.
Smart Images
Figure 2025079820000003 
Figure 2025079820000004 
Figure 2025079820000005
Abstract
Description
[Technical field]
[0001] background The present invention relates to apparatus, data structures, and computer-implemented methods for machine learning using semantic techniques. [Background technology]
[0002] Data labeling can be used in machine learning, where the labeled data is used to train a machine learning model to increase the knowledge of the machine learning model. Summary of the Invention [Problem to be solved by the invention]
[0003] Disclosure of the Invention An apparatus and method for machine learning provides data-efficient labeling and training. [Means for solving the problem]
[0004] A first computer-implemented method for machine learning includes providing a data structure of a database, the data structure including a set of nodes, in particular a set of entities, or a set of subjects and objects, the data structure including a set of relationships, in particular a set of edges, or a set of predicates, the data structure including a set of tuples, each tuple of the set of tuples including at least two nodes of the set of nodes and at least one relationship of the set of relationships, in particular two entities of the set of entities and one edge of the set of edges, or one subject and one object of the set of subjects and objects and one predicate of the set of predicates; the method including predicting a plurality of tuples depending on the data structure, each tuple of the plurality of tuples including at least two nodes of the set of nodes and at least one relationship of the set of relationships, in particular two entities of the set of entities and one edge of the set of edges, or one subject and one object of the set of subjects and objects and one predicate of the set of predicates; the method including predicting, for the plurality of tuples, an uncertainty as to whether each tuple of the plurality of tuples is classified as a member of the set of tuples, and selecting one tuple from the plurality of tuples depending on the uncertainty predicted for each tuple; obtaining a label indicating whether the selected tuple is classified as a member of the set of tuples; and adding the selected tuple to the set of tuples if the label indicates that the selected tuple is classified as a member of the set of tuples, or not adding the selected tuple to the set of tuples if the label indicates that the selected tuple is not classified as a member of the set of tuples. Predicting the plurality of tuples provides predicted links within the data structure. Labeling ensures a self-learning system that can process data from heterogeneous data sources for labeling the predicted links.For a predicted link, labels are requested that are more likely to improve performance on link prediction than those other than the predicted link. This is a data-efficient and resource-saving labeling approach because providing labels can be costly, e.g., due to the time it takes experts to label these labels.
[0005] The method may include predicting a number of tuples using a first model, the first model being trained to predict tuples, the tuples including at least two nodes of a set of nodes and at least one relationship of a set of relationships, depending on the data structure, in particular two entities of a set of entities and one edge of a set of edges, or one subject and one object of a set of subjects and objects and one predicate of a set of predicates, and training the first model depending on the selected tuples and labels. The first model is capable of fast predictions thanks to more effective link predictions. For example, links can be predicted quickly to enable search use cases in factories. A tuple is defined as an n-tuple, which is a finite sequence of a list of objects. Thus, the tuples may include, in particular, triples used in knowledge graphs, for example, during the course of semantic modeling.
[0006] The method may include: checking whether the obtained label is reliable, and if the label is reliable, training a first model depending on the selected tuple and the label, or not training the first model depending on the selected tuple and the label, or the method may include: checking whether the obtained label is valid, and if the label is valid, training a first model depending on the selected tuple and the label, or not training the first model depending on the selected tuple and the label, thereby preventing an incorrect label from being regarded as the ground truth.
[0007] The method may include obtaining the label from a second model configured to predict the label depending on the selected tuple, or obtaining the label from an expert or from an instance that can provide ground truth data, in particular from a test station or a diagnostic system or an expert system. The second model and the expert are exemplary data sources for automated labeling and semi-automated labeling, respectively.
[0008] Obtaining the label may include determining, in particular at the first device, a request for the label depending on the selected tuple, transmitting the request, in particular from the first device, to a second device, in which the user interface is included, or to a third device, in which the second model is included, and receiving, in particular at the first device, the label. The data source may be remote or local.
[0009] Obtaining the label may include sending a request and / or receiving the label over an electrical communications link that at least temporarily connects the devices and that is located at least partially outside the devices. The remote data source may be accessed regardless of the location of the remote data source.
[0010] Selecting a tuple may include determining a subset of the tuples that includes tuples that are not members of the set of tuples, and selecting the selected tuple depending on the predicted uncertainty for each tuple in the subset, which is an efficient approach to selecting new tuples for the set of tuples.
[0011] Selecting a tuple from the plurality of tuples may include determining, for the plurality of tuples, a measure indicative of the amount of information that can be obtained about the data structure from a label for each tuple of the plurality of tuples, and selecting the selected tuple depending on the uncertainty and / or the measure determined for the plurality of tuples, meaning that the predicted link that is most likely to contribute to the information gain about the links in the data structure is labeled.
[0012] Selecting a tuple from the plurality of tuples may include predicting a class of one of the plurality of classes for each triple, where the label indicates one of the plurality of classes, and the selected tuple is classified as a member of the set of tuples if the predicted class and the class indicated by the label match, and the selected tuple is not classified as a member of the set of tuples if the predicted class and the class indicated by the label do not match, thereby providing an efficient labeling technique in a multi-class scenario.
[0013] A second method for machine learning includes receiving a request for a label, the request including a tuple or an answer corresponding to a tuple of a data structure of a database, the label indicating whether the tuple is classified as a member of a set of tuples of the data structure of the database; requesting the label from an expert, in particular a human expert, or from an instance capable of providing ground truth data, in particular a test station or a diagnostic system or an expert system, using a user interface; and transmitting the label in response to the request. and labeling the data structure with a set of nodes, in particular a set of entities or a set of subjects and objects, the data structure with a set of relations, in particular a set of edges or a set of predicates, the data structure with a set of tuples, each tuple of the set of tuples including at least two nodes of the set of nodes and at least one relation of the set of relations, in particular two entities of the set of entities and one edge of the set of edges, or one subject and one object of the set of subjects and objects and one predicate of the set of predicates. This provides semi-automated labeling.
[0014] A third method for machine learning includes receiving a request for a label, the request including a tuple or an answer corresponding to a tuple of a data structure of a database, the label indicating whether the tuple is classified as a member of a set of tuples of the data structure of the database, requesting the label from the model, and sending the label in response to the request, the data structure including a set of nodes, in particular a set of entities, or a set of subjects and objects, the data structure including a set of relations, in particular a set of edges, or a set of predicates, the data structure including a set of tuples, each tuple of the set of tuples including at least two nodes of the set of nodes and at least one relation of the set of relations, in particular two entities of the set of entities and one edge of the set of edges, or a subject and one object of the set of subjects and objects and one predicate of the set of predicates, thereby providing automated labeling.
[0015] An apparatus for machine learning includes at least one processor and at least one memory, where the at least one processor is configured to execute instructions that, when executed by the at least one processor, cause the apparatus to perform a first method, a second method, or a third method, and the at least one memory is configured to store the instructions.
[0016] The computer program comprises computer readable instructions which, when executed by a computer, cause the computer to perform the first method, the second method or the third method.
[0017] The data structure comprises a set of nodes, in particular a set of entities or a set of subjects and objects; the data structure comprises a set of relations, in particular a set of edges or a set of predicates; the data structure comprises a set of tuples, each tuple of the set of tuples comprising at least two nodes of the set of nodes and at least one relation of the set of relations, in particular two entities of the set of entities and one edge of the set of edges, or one subject and one object of the set of subjects and objects and one predicate of the set of predicates; the data structure is configured to comprise a plurality of tuples predicted depending on the data structure, each tuple of the plurality of tuples comprising at least two nodes of the set of nodes and at least one relation of the set of relations, in particular two entities of the set of entities and one edge of the set of edges. or a subject and an object from a set of subjects and objects and a predicate from a set of predicates, the data structure being configured to include an uncertainty as to whether each tuple of the plurality of tuples is classified as a member of the set of tuples, the data structure being configured to include a tuple selected from the plurality of tuples depending on the predicted uncertainty for each tuple, the data structure being configured to include a label indicating whether the selected tuple is classified as a member of the set of tuples, and the data structure being configured to add the selected tuple to the set of tuples if the label indicates that the selected tuple is classified as a member of the set of tuples, or not add the selected tuple to the set of tuples if not, i.e. if the label indicates that the selected tuple is not classified as a member of the set of tuples.
[0018] Further embodiments will be readily derived from the following description and drawings. [Brief description of the drawings]
[0019] [Figure 1]FIG. 1 illustrates a schematic diagram of a first exemplary system for machine learning. [Diagram 2] FIG. 2 illustrates a schematic diagram of a second exemplary system for machine learning. [Diagram 3] FIG. 13 illustrates a schematic diagram of a third exemplary system for machine learning. [Figure 4] FIG. 1 illustrates a first method for machine learning. [Diagram 5] FIG. 2 illustrates a second method for machine learning. [Figure 6] FIG. 1 illustrates a third method for machine learning. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0020] FIG. 1 illustrates a schematic of a first exemplary system for machine learning.
[0021] The first exemplary system includes a first apparatus 100 for machine learning.
[0022] The first device 100 includes at least one processor 102 and at least one memory 104 .
[0023] The at least one memory 104 is configured to store a first model.
[0024] The at least one memory 104 is configured to store a database.
[0025] The first device 100 may be configured as a database management system for a database, for example configured to execute structured queries within the data structures of the database.
[0026] The data structure includes a set of nodes, a set of relationships, and a set of tuples, where each tuple in the set of tuples includes at least two nodes in the set of nodes and at least one relationship in the set of relationships.
[0027] The database includes a structure that is configured to provide access to a set of nodes, a set of relationships, and a set of tuples that are stored within the structure.
[0028] The data structure may include a knowledge graph. The set of nodes may be a set of entities specifically for the knowledge graph. The set of relationships may be a set of edges specifically for the knowledge graph. The set of tuples may be a set of tuples of the knowledge graph, each tuple of the set of tuples including two entities of the set of entities and one edge of the set of edges. The tuples in the knowledge graph may be triples.
[0029] The data structure may include statements about the semantic data. The set of nodes may be a set of subjects and objects for the statement. The set of relations may be a set of predicates for the statement. The set of tuples may be a set of tuples of the statement, each tuple in the set of tuples including one subject and one object from the set of nodes and one predicate from the set of predicates. The statements may be triples.
[0030] The set of nodes may be a set of subjects and objects for the knowledge graph. The set of relations may be a set of predicates for the knowledge graph. The set of tuples may be a set of tuples in the knowledge graph, where each tuple in the set of tuples includes one subject and one object from the set of nodes and one predicate from the set of predicates. The tuples in the knowledge graph may be triples.
[0031] The data structure comprises a set of relations, in particular a set of edges or a set of predicates, and the database comprises a set of tuples, each tuple of the set of tuples comprising at least two nodes of the set of nodes and at least one relationship of the set of relations, in particular two entities of the set of entities and one edge of the set of edges, or one subject and one object of the set of subjects and objects and one predicate of the set of predicates.
[0032] The first model is configured to provide an answer to a question. The apparatus 100 is configured to perform a structured query, for example, by providing a question as an input to the first model and obtaining an answer using the first model.
[0033] According to an example, the question includes a pattern and the answer satisfies the pattern. The pattern may include a tuple. For example, the pattern defines at least a portion of the answer or at least a portion of the tuples that correspond to the answer. The first model is configured, for example, to determine at least one answer or to determine at least one tuple that satisfies the pattern, or to output an indication that the answer satisfies the pattern or that there is no tuple that satisfies the pattern. The first model may be configured to output a tuple as an answer or to determine an answer corresponding to the tuple depending on the tuple.
[0034] The questions or answers may include text, video images, audio data, or alternatively may include numerical values, in particular predictions provided with uncertainties or margins of error or confidence intervals.
[0035] The first model may be configured to predict at least one tuple from the set of tuples depending on the question. The first model may be configured to predict at least one tuple from the set of tuples that satisfies the pattern depending on the pattern, meaning that predicting at least one tuple may include searching, finding or selecting a tuple in the set of tuples. The first model may be configured to predict at least one tuple that is not present in the set of tuples depending on the question. The first model may be configured to predict at least one tuple that satisfies the pattern and is not present in the set of tuples depending on the pattern.
[0036] The first model is configured to, e.g., question-dependent, e.g. pattern-dependent, predict a tuple that includes at least two nodes from the set of nodes and at least one relationship from the set of relationships.
[0037] The first model is configured, for example, to predict, depending on the question, for example, depending on the pattern, two entities of the set of entities and one edge of the set of edges.
[0038] The first model is configured to, for example, predict, in a question-dependent, e.g. pattern-dependent, manner, a subject and an object of a set of subjects and objects and a predicate of a set of predicates.
[0039] An example question or pattern for a predicted relationship between two nodes may include one given node and one given relationship. An answer may include the given node, the given relationship, and one predicted node. An answer may include one predicted node.
[0040] An example question or pattern for a predicted relationship between two nodes may include two given nodes. The answer may include these two given nodes and one predicted relationship. The answer may include one predicted relationship.
[0041] An example question or pattern for a predicted edge between two entities may include one given entity and one given edge. An answer may include the given entity, the given edge, and one predicted entity. An answer may include one predicted entity.
[0042] An example question or pattern for a predicted edge between two entities may include two given entities. The answer may include these two given entities and one predicted edge. The answer may include one predicted edge.
[0043] An example question or pattern for a predicted object may include one given subject and one given predicate. An answer may include the given subject, the given predicate, and the predicted object. An answer may include one predicted object.
[0044] An example question or pattern for a predicted subject may include one given object and one given predicate. An answer may include the predicted subject, the given predicate, and the given object. An answer may include the predicted subject.
[0045] The first model may be configured to provide answers to questions with an uncertainty that the answer is an answer to the question. The uncertainty is associated with each answer. The uncertainty may be provided with a confidence interval or a margin of error value, which may represent an absolute or relative margin of error value. The uncertainty associated with each answer quantifies the uncertainty of the first model with respect to the accuracy of the answer. The apparatus 100 is configured, for example, to use the first model to obtain the answers and the uncertainties associated with the answers.
[0046] The first model may be configured to represent uncertainty using a value between 0 and 1, with 1 indicating that the answer is known to be an answer to the question and 0 indicating that the answer is known not to be an answer to the question. Alternatively, uncertainty may be provided in the form of a normal distribution or a Student's t-distribution with no bounds or restrictions.
[0047] The first model may be configured to represent uncertainty using a value that indicates the probability that the answer is a true answer to the question as opposed to a false answer. In this example, an uncertainty of 0.5 indicates that it is unclear whether the answer is an answer to the question. The first model may be configured to represent uncertainty using a value between 0 and 1, different from 0,1.
[0048] The first model may be configured to predict, for at least one tuple, an uncertainty as to whether the tuple is classified as a member of the set of tuples.
[0049] The first model provides uncertainty that describes in a probabilistic way whether a tuple can be classified as a member of a set of tuples.
[0050] The first model may include an embedding model of a data structure, in particular an embedding model of a tuple of a set of tuples. The embedding may include a vector representation of the set of tuples, in particular a knowledge graph embedding.
[0051] The first model may include a link predictor.
[0052] The link predictor may be a graph neural network configured to predict tuples and respective uncertainties for each tuple depending on the similarity of vectors representing nodes, entities, or subjects and objects in the vector representation.
[0053] The link predictor may be a neural link predictor configured to predict tuples and respective uncertainties for each tuple. The neural link predictor may include, for example, a Bayesian model or a Bayesian graph neural network or a graph convolutional Gaussian process model. The graph convolutional Gaussian process model is described in Felix L. Opolka, Pietro Lio, “Graph Convolutional Gaussian Processes for Link Prediction”, 11 Feb 2020, https: / / arxiv.org / abs / 2002.04337 As described in, for example, a kernel that allows for interpolation between node neighborhoods of different sizes.
[0054] The apparatus 100 is configured to predict at least one tuple and the uncertainty using, for example, a Bayesian model or a Bayesian graph neural network or a graph convolutional Gaussian process.
[0055] The uncertainty predicted for a tuple may be, for example, the prediction uncertainty or prediction (co)variance of a Bayesian model for that tuple, or of a graph convolved Gaussian process.
[0056] The first device 100 may be configured for active learning.
[0057] The first device 100 may be configured to predict, using a neural link predictor, whether at least one tuple is classified as a member of the set of tuples.
[0058] The first device 100 is configured, for example, to add at least one tuple to a set of tuples if the at least one tuple is classified as a tuple of the set of tuples. The first device 100 is configured, for example, to store at least one tuple in a set of tuples if the at least one tuple is classified as a tuple of the set of tuples in the data structure and / or database. The first device 100 is configured, for example, to not add at least one tuple to a set of tuples if the at least one tuple is not classified as a tuple of the set of tuples.
[0059] The first device 100 may be configured to predict the plurality of tuples using a neural link predictor.
[0060] The first device 100 may be configured to predict uncertainties for a number of tuples using a first model, in particular using a neural link predictor.
[0061] The first device 100 may be configured to determine whether each tuple of the plurality of tuples is classified as a member of the set of tuples depending on the uncertainty.
[0062] The first device 100 may be configured to select a tuple from the plurality of tuples as a tuple to be classified as a member of the set of tuples if, for example, the uncertainty is lower than the first uncertainty, e.g., if the uncertainty value is 1, or if it is greater than or equal to a threshold value, e.g., 0.9.
[0063] The first device 100 may be configured to, for example, determine not to classify a tuple from the plurality of tuples as a member of the set of tuples if the uncertainty is lower than the second uncertainty, for example if the uncertainty value is 0, or if it is below a threshold value, for example 0.1.
[0064] The first device 100 may, for example, be configured to determine that it is uncertain whether a tuple from the plurality of tuples is classified as a member of the set of tuples when the uncertainty is between a first uncertainty and a second uncertainty, for example between both thresholds.
[0065] Labeling a tuple is costly since it is unknown whether the tuple will be classified as a member of the set of tuples. The first device 100 may be configured to label only selected tuples that are likely to reduce the uncertainty of the first model.
[0066] The first device 100 may be configured to select a selected tuple from the multiple tuples depending on the predicted uncertainty for each tuple, and thereby determine whether the selected tuple is classified as a member of the set of tuples.
[0067] The apparatus 100 may be configured to determine a ranking of the tuples using an acquisition function, whereby the tuples are ranked depending on the uncertainty associated with each tuple. The acquisition function may take into account a measure indicating the amount of information that can be obtained about the data structure from a label for each tuple of the plurality of tuples. The measure may be mutual information, MI or entropy, in particular Shannon entropy. The measure may for example depend on the plurality of tuples.
[0068] Selecting a tuple from the plurality of tuples depending on the uncertainty may include selecting a tuple associated with a highest uncertainty depending on the uncertainty as the selected tuple. Selecting a tuple from the plurality of tuples depending on the measure may include selecting a tuple associated with a maximum measure as the selected tuple.
[0069] The apparatus 100 may be configured to select the highest ranked tuple as the selected tuple.
[0070] To determine whether the selected tuple is a member of the set of tuples, the first device 100 may include an interface 106 configured to obtain a label 108 indicating whether the selected tuple is classified as a member of the set of tuples.
[0071] The interface 106 may be configured to send a request 110 for the label 108 .
[0072] The request 110 may include, for example, a selected tuple or an answer corresponding to the selected tuple.
[0073] The first device 100 may be configured to add the selected tuple to the set of tuples if the label 108 indicates that the selected tuple is a member of the set of tuples. The first device 100 may be configured to not add the selected tuple to the set of tuples otherwise, i.e., if the label 108 indicates that the selected tuple is not a member of the set of tuples.
[0074] The first device 100 may be configured to train the first model and / or the neural link predictor depending on the selected tuples or answers corresponding to the selected tuples and the labels 108.
[0075] The first model may be configured to predict one of the multiple classes for a tuple of the multiple tuples. Selecting a tuple from the multiple tuples may include predicting one of the multiple classes for each triple.
[0076] The label 108 may indicate one class of multiple classes.
[0077] According to one example, a selected tuple is classified as a member of the set of tuples if the predicted class and the class indicated by the label 108 match.
[0078] According to one example, a selected tuple is not classified as a member of the set of tuples if there is a mismatch between the predicted class and the class indicated by label 108 .
[0079] According to a first exemplary system, the first device 100 may be configured to obtain labels 108 from a second device 112 for machine learning.
[0080] The second device 112 is configured to request the labels 108 from an expert. The expert may be a human or a machine-based instance that can provide ground truth data, such as a test station or a diagnostic system or an expert system. Additionally, the expert may be implemented as a combination of machine labeling and human labeling. For example, the expert may be implemented using multiple instances, which may be human or machine-based, by majority voting, or by utilizing the law of large numbers.
[0081] The second device 112 is, for example, a computer terminal, a personal computer, a mobile phone, in particular a smartphone, or a handheld computing device, for example a tablet computer.
[0082] The second device 112 includes at least one processor 114 and at least one memory 116. The second device 112 includes a first interface 118 configured to receive the request 110 and transmit the label 108 in response to the request 110. The second device 112 includes a user interface 120. The user interface 120 includes, for example, a display configured to display the request to an expert 110. The user interface 120 includes, for example, an input configured to detect the expert's label 108.
[0083] According to a first exemplary system, a first device 100 may be configured to receive a question 122 and determine an answer 124 to the question.
[0084] The interface 106 of the first device 100 is configured to, for example, receive a question 122. According to a first exemplary system, the user interface 120 may be configured to detect the question 122 from a user input, for example a voice input or a text input, and the interface 118 of the second device 112 is configured to transmit the question 122. The question 122 may be received by the interface 106 of the first device 100 from another device external to the first device 100.
[0085] The first device 100 is configured to map questions 122 to answers 124 using a first model.
[0086] The interface 106 of the first device 100 is configured to transmit the answer 124. According to a first exemplary system, the interface 118 of the second device 112 is configured to receive the answer 124, and the user interface 120 is configured to output, e.g., display or read, the answer to the user. The answer 124 may be transmitted by the interface 106 to another device external to the first device 100.
[0087] The second device 112 is external to the first device 100. The second device 112 may be remote from the first device 100. According to the first exemplary system, a question is transmitted by the first device 100 to the second device 112, in particular via a telecommunications link. According to the first exemplary system, an answer is transmitted by the second device 112 to the first device 100, in particular via a telecommunications link.
[0088] FIG. 2 illustrates a schematic of a second exemplary system for machine learning.
[0089] The second exemplary system includes a first device 100 .
[0090] The second exemplary system includes a third device 126 for machine learning.
[0091] The third device 126 is configured to request the label 108 from a second model 128. The second model 128 may be an external model, for example an expert system.
[0092] The third device 126 includes at least one processor 130 and at least one memory 132. The third device 126 includes an interface 134 configured to receive the request 110 and to transmit the label 108 in response to the request 110. The third device 126 includes a second model 128. The second model 128 is configured to map the request 110 to the label 108.
[0093] The third device 126 is external to the first device 100. The third device 126 may be remote from the first device 100. According to the second exemplary system, a question is transmitted by the first device 100 to the third device 126, in particular via a telecommunications link. According to the second exemplary system, an answer to the question is transmitted by the third device 126 to the first device 100, in particular via a telecommunications link.
[0094] FIG. 3 illustrates generally a third exemplary system for machine learning.
[0095] The third exemplary system includes a first device 100. According to the third exemplary system, the first device 100 is configured to internally determine an answer to a question.
[0096] According to a third exemplary system, the first device 100 may include a second model 128 and / or a user interface 120 in addition to or instead of the first interface 106 of the first device 100.
[0097] The at least one processor 102 of the first device 100 is configured to execute first instructions that, when executed by the at least one processor 102 of the first device 100, cause the first device 100 to implement a first method for machine learning. The at least one memory 104 of the first device 100 is configured to store the first instructions.
[0098] FIG. 4 shows a flow chart including steps of the first method.
[0099] The first method includes step 402 .
[0100] Step 402 involves providing a data structure for a database.
[0101] The data structure includes a set of nodes, in particular a set of entities or a set of subjects and objects.
[0102] The data structure includes a set of relations, in particular a set of edges, or a set of predicates.
[0103] The data structure contains a collection of tuples.
[0104] Each tuple in the set of tuples includes at least two nodes in the set of nodes and at least one relationship in the set of relationships.
[0105] This means that a tuple in a data structure containing a set of entities and a set of edges contains two entities from the set of entities and one edge from the set of edges.
[0106] This means that a tuple of a data structure containing a set of subjects and objects contains one subject and one object from the set of subjects and objects, and one predicate from the set of predicates.
[0107] The first method includes step 404 .
[0108] Step 404 includes predicting multiple tuples depending on the data structure.
[0109] Each tuple of the plurality of tuples includes at least two nodes of the set of nodes and at least one relationship of the set of relationships.
[0110] This means that for a data structure containing a set of entities and a set of edges, a tuple of tuples predicted will contain two entities from the set of entities and one edge from the set of edges.
[0111] This means that a tuple of tuples predicted for a data structure containing a set of subjects and objects contains a subject and an object from the set of subjects and objects, and a predicate from the set of predicates.
[0112] According to one example, the tuples are predicted using a first model.
[0113] The first method includes step 406 .
[0114] Step 406 includes predicting, for the plurality of tuples, an uncertainty as to whether each tuple of the plurality of tuples is classified as a member of the set of tuples.
[0115] According to one example, the uncertainty is predicted using a first model.
[0116] The first method includes step 408 .
[0117] Step 408 involves selecting a tuple from the multiple tuples depending on the predicted uncertainty for each tuple.
[0118] Selecting the tuples may include determining a subset T\T1 of multiple tuples T that are not members of the set T1 of tuples.
[0119] Selecting the tuples may include selecting the selected tuples depending on the predicted uncertainty for each tuple in the subset T\T1.
[0120] Additionally or alternatively, selecting a tuple from the plurality of tuples may include determining, for the plurality of tuples, a measure indicative of an amount of information that can be obtained about the data structure from a label for each tuple in the plurality of tuples.
[0121] For example, an uncertainty or measure or a function thereof for each tuple of the plurality of tuples is associated with each tuple. For example, the tuples in the plurality of tuples are ranked by a product value. For example, the tuple associated with the maximum product value is selected as the selected tuple.
[0122] Selecting a tuple from the plurality of tuples may include predicting one class of the plurality of classes for each triple.
[0123] The label 108 may indicate one class of multiple classes.
[0124] According to one example, a selected tuple is classified as a member of the set of tuples if the predicted class and the class indicated by the label 108 match.
[0125] According to one example, a selected tuple is not classified as a member of the set of tuples if there is a mismatch between the predicted class and the class indicated by label 108 .
[0126] The first method includes step 410 .
[0127] Step 410 involves obtaining a label 108 that indicates whether the selected tuple is classified as a member of the set of tuples.
[0128] The first method may include obtaining the labels 108 from a second model 128 or an expert.
[0129] Obtaining the label 108 may include determining a request 110 for the label 108 depending on the selected tuple. Obtaining the label 108 may include sending the request 110 from the first device 100 to the second device 112 or the third device 126, among others.
[0130] Obtaining the label 108 may include, inter alia, receiving the label 108 at the first device 100 .
[0131] Obtaining the label 108 may include sending the request 110 and / or receiving the label 108 over an electrical communication link. The electrical communication link may in particular at least temporarily connect the first device 100 and the second device 112 or may connect the first device 100 and the third device 126. The electrical communication link may be located at least partially outside the first device, the second device 112, and / or the third device 126.
[0132] The first method includes step 412 .
[0133] Step 412 involves determining whether the label 108 indicates that the selected tuple is classified as a member of the set of tuples.
[0134] If the label 108 indicates that the selected tuple is classified as a member of the set of tuples, then step 414 is performed. Otherwise, i.e., if the label 108 indicates that the selected tuple is not classified as a member of the set of tuples, then step 404 can be performed to predict another different tuple, or the method can end, for example, if no new tuples are found that are classified as a member of the set of tuples.
[0135] Step 414 includes adding the selected tuples to a collection of tuples, which may be stored in a data structure and / or a database.
[0136] Then, step 416 is performed.
[0137] Step 416 involves training a first model, in particular a neural link predictor, depending on the selected tuples and the labels 108 .
[0138] An exemplary algorithm for a knowledge graph, which includes triples of subject, predicate, and object, and in which a link predictor provides binary uncertainties, e.g., true, false, includes generating, for each triple in the knowledge graph, a probability P(triple=true) that it is true, meaning it is part of the knowledge graph, such that P(triple=false)=1-P(triple=true).
[0139] Let T be the set of theoretically possible triples for the knowledge graph, T1 be a subset of T, i.e., the set of triples that are the basis of the knowledge graph, and T2 be a subset of T that contains triples that are known not to be classified as triples in the set of triples, e.g., P(triple=false)=1, where P is the uncertainty provided by the link predictor.
[0140]
number
[0141] The argmin acquisition function covers how certain the link predictor is with respect to triples that are classified as triples in the set of triples underlying the knowledge graph.
[0142] Optionally, the acquisition function not only covers how robust the link predictor is, but also how representative the expected label l is for other search tasks and how many other network participants or link predictions will benefit from this label.
[0143] To achieve this, we change line 2 of the algorithm to:
number
[0144] Let us consider an element e∈E in the set of elements E to belong to exactly one class c in the set of classes C. i An exemplary algorithm for knowledge graphs in a multi-class scenario, such as a set of classes that can be part of a set of classes, is to calculate the uncertainty P(e in c i ), i.e., class c i If ∈C, then element e is in class c i This results in a discrete probability distribution over the classes. * A measure for determining may be the entropy over this probability distribution: e * =argmax Entropy(P(e in c i ))
[0145] Alternatively, element e * Alternatively, or in addition to entropy, the class c i The uncertainty P(e * in c i ), the element e with the smallest maximum uncertainty * may be selected.
[0146] Mutual information may be used instead of or in addition to entropy or minimum maximum uncertainty.
[0147] Possible class c in the use case i may be a location or identifier, for example the name of a factory, to which characteristics of a particular machine or equipment may be assigned.
[0148] A particular machine may be a machine from a set of machines configured to perform at least one operation from a set of operations. The machine may be configured to perform at least one operation from a set of operations. The machine is implemented at a station from a set of stations configured to implement at least one machine from the set of machines. At least some of the machines may be configured to perform only a subset of the operations. At least some of the stations may be configured to implement only a subset of the machines. The machine may belong to a factory from a set of factories.
[0149] The set of predicates in the knowledge graph may include the predicates "belongsTo", "executesProcess", and "isImplementedBy". The set of subjects and objects in the knowledge graph may include processes in a set of processes, machines in a set of machines, stations in a set of stations, and factories in a set of factories.
[0150] Multi-class classification problems that can be solved using algorithms for knowledge graphs in multi-class scenarios include: Station executes Process, Machine isImplementedBy Station where “Station”, “Process”, and “Machine” are variables of a multi-class classification problem for subjects and / or objects of a set of subjects and objects for a knowledge graph from a set of stations, a set of processes, and a set of machines, respectively.
[0151] A solution to a multi-class classification problem by way of example provides for the allocation of processes in a set of processes to machines in a set of machines.
[0152] Before solving a multi-class classification problem, conditioning may be performed. Conditioning is performed by combining the predicate “belongsTo” and the class classification problem: Machine belongsTo Plant and determining heterogeneous subgraphs of the knowledge graph depending on the set of machine and the set of plants, where "Machine" and "Plant" are variables of a multi-class classification problem for subjects and / or objects of a set of subjects and objects for the knowledge graph from the set of machines and the set of plants, respectively.
[0153] A knowledge graph may be used to answer questions about machines. For example, a knowledge graph can be used to answer the question "To which plant does machine YY belong to?". For example, the set of machines includes machine YY. The question includes machine YY as a subject and "belongsTo" as a predicate. The answer to the question includes the factories in the set of factories that solve this multi-class classification problem.
[0154] Especially during self-learning, steps 404 through 416 can be repeated iteratively to retrain and collect new labels.
[0155] Semi-automated labelling by a human expert can be performed in a decentralised manner, for example using the second device 112.
[0156] The at least one processor 114 of the second device 112 is configured to execute second instructions that, when executed by the at least one processor 114 of the second device 112, cause the second device 112 to implement a second method for machine learning. The at least one memory 116 of the second device 112 is configured to store the second instructions.
[0157] FIG. 5 shows a flow chart including steps of the second method.
[0158] The second method includes step 502 .
[0159] Step 502 involves receiving a request 110 .
[0160] Then, step 504 is executed.
[0161] Step 504 involves requesting labels 108 from an expert, particularly a human expert, using the user interface 120 .
[0162] For example, the request 110 is presented to the expert, particularly on a display. For example, the expert's input is detected, particularly by an input portion of the user interface 120. According to one example, the expert's input includes a label 108. According to one example, the expert's input is mapped to the label 108. The label 108 can indicate true or false. The label 108 can indicate a class among multiple classes.
[0163] Then, step 506 is executed.
[0164] Step 506 includes transmitting the label 108 in response to the request 110 .
[0165] Automated labeling with the second model 128 can be performed using a third device 126 .
[0166] At least one processor 130 of the third device 126 is configured to execute a third instruction for causing the third device 126 to implement a third method for machine learning when executed by the at least one processor 130 of the third device 126. At least one memory 132 of the third device 126 is configured to store the third instruction.
[0167] FIG. 6 shows a flowchart including steps of the third method.
[0168] The third method includes step 602.
[0169] Step 602 includes receiving a request 110.
[0170] Thereafter, step 604 is executed.
[0171] Step 604 includes requesting a label 108 from the second model 128.
[0172] According to an example, the output of the second model 128 includes the label 108. According to an example, the output of the second model 128 is mapped to the label 108. The label 108 can indicate true or false. The label 108 can indicate a class among a plurality of classes.
[0173] Thereafter, step 606 is executed.
[0174] Step 606 includes transmitting the label 108 in response to the request 110.
[0175] Optionally, the labeling may include checking whether the answer of the expert or second model 128 is reliable in a validation step. The validation step may include evaluating the historical reliability of the expert or second model 128 or determining whether the answer of the expert or second model 128 is within an expected range. This range defines, for example, conditions that must be met by a valid answer. An answer that does not meet the conditions is out of range and is considered invalid.
[0176] The first model may be configured to determine ranges for tuples predicted by the first model. The link predictor may be configured to determine ranges for tuples predicted by the link predictor.
[0177] The range may be determined using a first model or a link predictor.
Claims
1. 1. A first computer-implemented method for machine learning, comprising: The method includes providing 402 a database data structure; said data structure comprising a set of nodes, in particular a set of entities or a set of subjects and objects; the data structure comprises a set of relations, in particular a set of edges or a set of predicates, the data structure includes a collection of tuples; Each tuple of the set of tuples comprises at least two nodes of the set of nodes and at least one relationship of the set of relationships, in particular two entities of the set of entities and one edge of the set of edges, or a subject and an object of the set of subjects and objects and a predicate of the set of predicates, The method includes predicting tuples depending on the data structure (404); Each tuple of the plurality of tuples includes at least two nodes of the set of nodes and at least one relationship of the set of relationships, in particular two entities of the set of entities and one edge of the set of edges, or a subject and an object of the set of subjects and objects and a predicate of the set of predicates; The method comprises: predicting (406) for the plurality of tuples an uncertainty as to whether each tuple of the plurality of tuples is classified as a member of the set of tuples; selecting (408) a tuple from the plurality of tuples depending on the uncertainty predicted for each of the tuples; obtaining (410) a label (108) indicating whether the selected tuple is classified as a member of the set of tuples; adding (414) the selected tuple to the set of tuples if the label (108) indicates (412) that the selected tuple is classified as a member of the set of tuples, or not adding the selected tuple to the set of tuples if not, i.e., if the label (108) indicates that the selected tuple is not classified as a member of the set of tuples; Including, 1. A first computer-implemented method for machine learning, comprising:
2. The method comprises: predicting the plurality of tuples using a first model (404), the first model being trained to predict a tuple, the tuple including, depending on the data structure, at least two nodes of the set of nodes and at least one relationship of the set of relationships, in particular two entities of the set of entities and one edge of the set of edges, or a subject and an object of the set of subjects and objects and a predicate of a set of predicates (404); training (416) the first model dependent on the selected tuples and the labels (108); The method of claim 1 , comprising:
3. The method comprises: checking whether the obtained label (108) is trustworthy; If the label (108) is reliable, training (416) the first model depending on the selected tuple and the label (108), or not training the first model depending on the selected tuple and the label (108); Including, Or, The method comprises: checking whether the obtained label (108) is valid; If the label (108) is valid, training (416) the first model depending on the selected tuple and the label (108), or not training the first model depending on the selected tuple and the label (108); The method of claim 2 , comprising:
4. The method comprises: obtaining (410) the label (108) from a second model (128) configured to predict the label (108) dependent on the selected tuples; Or, Obtaining (410) said labels (108) from an expert or from an instance capable of providing ground truth data, in particular from a test station or a diagnostic system or an expert system. The method of any one of claims 1 to 3, comprising:
5. Obtaining (410) the label (108) comprises: determining, in particular in the first device (100), a request (110) for said label (108) depending on said selected tuple; sending said request (110), in particular from said first device (100), to a second device (112) comprising a user interface (120) or to a third device (126) comprising said second model (128); In particular, receiving, in said first device (100), said label (108); The method of claim 4 , comprising:
6. Obtaining (410) the label (108) includes transmitting the request (110) and / or receiving the label (108) over an electrical communication link; the electrical communication link at least temporarily connects the devices (100, 112; 126) and is located at least partially outside the devices (100, 112; 126); The method according to claim 5.
7. Selecting 408 the tuple may include: determining a subset of the plurality of tuples that includes tuples that are not members of the set of tuples; selecting the selected tuples in dependence on the uncertainty predicted for each tuple in the subset; The method of any one of claims 1 to 6, comprising:
8. Selecting 408 a tuple from the plurality of tuples includes: determining, for the plurality of tuples, a measure (MI) indicative of the amount of information that can be obtained about the data structure from a label for the respective tuple of the plurality of tuples; selecting the selected tuples depending on the uncertainty and / or the measure determined for the plurality of tuples; The method of any one of claims 1 to 7, comprising:
9. Selecting 408 a tuple from the plurality of tuples includes predicting one class of a plurality of classes for each triple; the label (108) indicates one of the classes; the selected tuple is classified as a member of the set of tuples if the predicted class matches the class indicated by the label (108); the selected tuple is not classified as a member of the set of tuples if there is a mismatch between the predicted class and the class indicated by the label (108).
9. The method according to any one of claims 1 to 8.
10. A second computer-implemented method for machine learning, comprising: The method comprises: receiving (502) a request (110) for a label (108), the request (110) including a tuple or an answer corresponding to a tuple of a data structure of a database, the label (108) indicating whether the tuple is classified as a member of a set of tuples of the data structure of the database; - requesting (504) said labels (108) from an expert, in particular a human expert, or from an instance capable of providing ground truth data, in particular a test station or a diagnostic system or an expert system, using a user interface (120); transmitting (506) the label (108) in response to the request (110); Including, said data structure comprising a set of nodes, in particular a set of entities or a set of subjects and objects; the data structure comprises a set of relations, in particular a set of edges or a set of predicates, the data structure includes a collection of tuples; Each tuple of the set of tuples comprises at least two nodes of the set of nodes and at least one relationship of the set of relationships, in particular two entities of the set of entities and one edge of the set of edges, or a subject and an object of the set of subjects and objects and a predicate of the set of predicates. A second computer-implemented method for machine learning, comprising:
11. A third computer-implemented method for machine learning, comprising: The method comprises: receiving (602) a request (110) for a label (108), the request (110) including a tuple or an answer corresponding to a tuple of a data structure of a database, the label (108) indicating whether the tuple is classified as a member of a set of tuples of the data structure of the database; requesting (604) the label (108) from a model (128); transmitting (606) the label (108) in response to the request (110); Including, said data structure comprising a set of nodes, in particular a set of entities or a set of subjects and objects; the data structure comprises a set of relations, in particular a set of edges or a set of predicates, the data structure includes a collection of tuples; Each tuple of the set of tuples comprises at least two nodes of the set of nodes and at least one relationship of the set of relationships, in particular two entities of the set of entities and one edge of the set of edges, or a subject and an object of the set of subjects and objects and a predicate of the set of predicates. A third computer-implemented method for machine learning, comprising:
12. An apparatus (100; 112; 126) for machine learning, comprising: The device (100; 112; 126) comprises: At least one processor (102; 114; 130); At least one memory (104; 116; 132); Equipped with The at least one processor (102; 114; 130) is configured to execute instructions which, when executed by the at least one processor (102; 114; 130), cause the device (100; 112; 126) to perform a method according to any one of claims 1 to 10, said at least one memory (104; 116; 132) configured to store said instructions; An apparatus (100; 112; 126) for machine learning.
13. A computer program comprising: The computer program comprises computer readable instructions which, when executed by a computer, cause the computer to carry out the method according to any one of claims 1 to 11. A computer program comprising:
14. 1. A data structure comprising: said data structure comprising a set of nodes, in particular a set of entities or a set of subjects and objects; the data structure comprises a set of relations, in particular a set of edges or a set of predicates, the data structure includes a collection of tuples; Each tuple of the set of tuples comprises at least two nodes of the set of nodes and at least one relationship of the set of relationships, in particular two entities of the set of entities and one edge of the set of edges, or a subject and an object of the set of subjects and objects and a predicate of the set of predicates, the data structure is configured to include a number of tuples predicted depending on the data structure; Each tuple of the plurality of tuples includes at least two nodes of the set of nodes and at least one relationship of the set of relationships, in particular two entities of the set of entities and one edge of the set of edges, or a subject and an object of the set of subjects and objects and a predicate of the set of predicates; the data structure is configured to include an uncertainty regarding whether each tuple of the plurality of tuples is classified as a member of the set of tuples; the data structure is configured to include a tuple selected from the plurality of tuples depending on the uncertainty predicted for each tuple; the data structure is configured to include a label (108) indicating whether the selected tuple is classified as a member of the set of tuples; the data structure is configured to add the selected tuple to the set of tuples if the label (108) indicates that the selected tuple is classified as a member of the set of tuples, or to not add the selected tuple to the set of tuples otherwise, i.e., if the label (108) indicates that the selected tuple is not classified as a member of the set of tuples.
1. A data structure comprising: