Apparatus and method for determining a knowledge graph

Abstract

The invention relates to a device and a computer-implemented method for determining a knowledge graph, comprising determining (404) an embedding of a label sequence of an instance; determining (406) a first classification of the embedding at a first classifier; determining (410) whether the first classification satisfies a first condition; when the first classification satisfies the first condition, adding (412) a first link between a first node of the knowledge graph representing the instance and a node of the knowledge graph representing the first classification to the knowledge graph, otherwise not adding the first link.

Description

Technical Field

[0001] This invention relates to apparatus and methods for determining knowledge graphs, particularly for determining knowledge graphs based on solutions to classification problems. Summary of the Invention

[0002] A computer-implemented method for determining a knowledge graph includes: determining the embedding of a sequence of tags for instances; determining a first category of the embedding at a first classifier; determining whether the first category satisfies a first condition; and when the first category satisfies the first condition, adding a first link between a first node representing the instance in the knowledge graph and a node representing the first category in the knowledge graph, otherwise not adding the first link. The first classifier is associated with nodes representing tags in a tag tree of the knowledge graph. The first category may be that the instance belongs to that tag or that the instance does not belong to that tag.

[0003] Preferably, the method includes determining a second classification at a second classifier; determining whether the second classification satisfies a second condition; when the second classification satisfies the second condition, adding a second link to the knowledge graph between a node representing the instance in the knowledge graph and a second node representing the second classification in the knowledge graph; otherwise, not adding the second link. In a multi-task setting, the main task is decomposed into a set of tasks, and the classifier outputs a prediction separately for each task.

[0004] The method may include providing the embedding to the first classifier; and providing the embedding and / or the hidden state of the first classifier resulting from the provided embedding as input to the second classifier. Classifiers at different levels in the hierarchy are assigned to different levels in the label hierarchy of the label tree. The links between parent and child nodes in the knowledge graph are represented by the hierarchical structure of classifiers. The classification or hidden state of the classifier representing the parent can be used as input to the classifiers representing its children.

[0005] The method may include determining the first link from the node representing the instance to the first node representing the first category when the second classification satisfies a condition. When the second classifier is at a lower level in the classifier hierarchy than the first classifier, the predictions integrated by the classifier are used for hierarchical prediction patterns or post-processing predictions are used to make the hierarchical predictions consistent, for example, adding the relevant parent node to the knowledge graph if a relevant parent node has not yet been predicted but a child node has been predicted, or deleting a child label if a parent node has not yet been predicted.

[0006] The token sequence of the instance can be determined. A tokenizer can be used to determine the sequence. The instance may include numeric text data.

[0007] Preferably, the first classification and / or the second classification is a binary classification.

[0008] The method may include providing nodes representing a tag tree to the knowledge graph, and adding multiple links to the knowledge graph to the nodes representing tags of the instance. A single tag or multiple tags can be assigned to the instance in this way.

[0009] Preferably, the method includes using the first classification and / or the second classification to determine whether the instance belongs to a category represented by a node in a label tree.

[0010] The method preferably includes assigning the first category to a label different from the second category. Therefore, different tasks are addressed in a multi-tasking setting.

[0011] For training, the method includes providing data points comprising labels for the first classification and / or labels for the second classification, and training the first classifier and / or the second classifier based on the data points. This means training different classifiers to make different decisions. This allows for solving different tasks in a multi-task setting. This combines the advantages of hierarchical classification algorithms with the advantages of using a pre-trained neural language model based on a transformer for classification, in which training classifiers focused on different decisions in a hierarchical classification algorithm optimally utilizes the training data. Since all classifiers share a single transformer model and different classifiers are trained in a multi-task setting, computational feasibility is ensured even in settings with limited available computing power.

[0012] Preferably, the method includes providing a model, particularly a neural network, and training the model to determine the first classification or the second classification. Therefore, for each label, a portion of the neural network representing a classifier is trained.

[0013] Training the model may include determining a loss from the output of the first classifier and a loss from the output of the second classifier, and backpropagating the losses to train the weights in the first and second classifiers based on the two losses, or training the first classifier based on the first loss and independent of the second loss, and training the second classifier based on the second loss and independent of the first loss.

[0014] Preferably, the method includes determining a classification for the input of the model trained in this manner using a classifier located at a position in the classifier hierarchy of the model, and assigning the classification to a label corresponding to the position in the hierarchy of the label tree. The classifier ensemble in the model can be used for hierarchical prediction. This ensures consistency of labels across the tree.

[0015] The device used to determine the knowledge graph is configured to perform the steps of the method. Attached Figure Description

[0016] Other advantageous embodiments can be readily derived from the following description and accompanying drawings. (See the accompanying drawings.) Figure 1 The device used to determine the knowledge graph is described. Figure 2 A tag tree was depicted. Figure 3-A depicts a schematic diagram of the classification according to the first aspect. Figure 3-B illustrates a classification based on the second aspect. Figure 4 The steps in the method for determining the knowledge graph are described. Figure 5 The steps in the method for training a model for determining knowledge graphs are described. Detailed Implementation

[0017] The device 100 used to determine knowledge graph 102 in this example includes at least one processor 104, at least one memory 106, and an interface 108. Interface 108 is adapted to receive input data from knowledge graph 102 and output information about the input data. Data link 110 at least temporarily connects at least one processor 104, interface 108, and at least one memory 106.

[0018] Device 100 is configured to perform the steps of the following method.

[0019] At least one memory 106 may include a computer program. The computer program may include computer-readable instructions that, when executed by at least one processor 104, cause the device 100 to perform the steps in the method.

[0020] In the example below, knowledge graph 102 represents a solution to a classification problem determined by hierarchical multi-label classification. Hierarchical multi-label classification is a type of classification problem in which entries (such as instances of text or digital images) are classified into hierarchical labels, and multiple labels can be assigned to each hierarchical level.

[0021] The instance in this example is stored in at least one memory 106 or received via interface 108. The instance may include digital text data.

[0022] In the aspects described below, a binary classifier is trained for either the leaf nodes or non-leaf nodes of the label tree. Figure 2 An exemplary tag tree 200 constituting a hierarchical structure is depicted. This hierarchy has a highest level, referred to as the root level 202, and has a first level 204, a second level 206, and a third level 208 in descending hierarchical order.

[0023] The root level in this example includes the node labeled Root. The first level 204 in this example includes non-leaf nodes labeled A and B. Both are directly linked to the node labeled Root. They are not directly linked to each other. In one respect, every node in the label tree has exactly one parent node.

[0024] In this example, level 206 includes a non-leaf node labeled A43 and a non-leaf node labeled A44. Both are directly linked to node A. Neither is linked to node B, nor to the node marked as Root, and they are not directly linked to each other.

[0025] In this example, level 206 includes a non-leaf node labeled B41 and a non-leaf node labeled B23. Both are directly linked to node B. Neither is linked to node A, nor to the node marked as Root, and they are not directly linked to each other.

[0026] In this example, level 3 208 includes leaf nodes for label A43B and label A43C. Both are directly linked to node A43. Neither is linked to any other node.

[0027] In this example, level 3 208 includes leaf nodes for label A44B and label A44C. Both are directly linked to node A44. Neither is linked to any other node.

[0028] In this example, level 3 208 includes leaf nodes for label B41J and label B41F. Both are directly linked to node B41. Neither is linked to any other node.

[0029] In this example, level 3 208 includes leaf nodes for label B23D and label B23Q. Both are directly linked to node B23. Neither is linked to any other node.

[0030] A portion of the classifier ensemble 300 is depicted in the schematic diagram of the classification according to the first aspect in Figure 3-A, where a classifier is trained for each node of the tree. Figure 3-B depicts the classification according to the second aspect.

[0031] More specifically, Figures 3-A and 3-B depict an artificial neural network architecture that has the following features: Figure 2 The tree includes the classifier of labels in the branches of node A. The integration 300 according to the first aspect is described below with reference to Figure 3-A.

[0032] The input for Integration 300 can be an instance that includes a sequence of tags. The input can include the title or abstract of a patent document. In this example, the tags are based on patent classification.

[0033] In the first aspect depicted in Figure 3-A, the input is raw text 301 without any preprocessing. An appropriate word segmenter 302 (e.g., word-slice segmentation) segments the text 301 into a sequence of tokens. In this example, the sequence of tokens is one-hot encoded to form an input vector t1, t2, t3, ... tn for each token, prepared by using token identifiers 303 corresponding to the words as preparation for a transformer layer 304 hereinafter referred to as the transformer. Input vectors for specific classification tokens are also provided and labeled in Figure 3-A as follows. <cls>.

[0034] Transformer 304 in this example is a neural language model based on a transformer, such as a pre-trained neural language model like BERT or RoBERTa.

[0035] In transformer 304, the input vector represents the special classification label. <cls>The input vectors t1, t2, t3, ..., tn, along with the one-hot encoded tag sequence, are processed by a pre-trained transformer, i.e., a language model, to generate embedding 306. Embedding 306 includes tag embeddings 306-1, 306-2, ..., 306-n for the input vectors t1, t2, t3, ..., tn. Embedding 306 also includes embeddings 307 for specific classification tags, such as CLS used in BERT.

[0036] In one aspect, BERT outputs a special sequence of embeddings, the first of which is an embedding 307 corresponding to the classification label and can be used as a representative of the entire input sequence. Furthermore, BERT outputs an embedding for each label. In this aspect, CLS embeddings are used to form a single embedding that serves as input to the classifier and / or the rest of the neural network.

[0037] The example is explained below based on the embedding using CLS tags as the aggregation vector 308.

[0038] The transformer's output embedding 306 forms a matrix. The dimension of the matrix depends on the maximum number of tokens. With M tokens in the text and an embedding size of E, the transformer 304 generates a token tensor of dimension M×E. The first vector within this token tensor corresponds to the embedding 307 of the CLS tokens and can be considered as the embedding of the entire input sequence; that is, this first vector can be used as the aggregate embedding 308.

[0039] Aggregate embedding 308 can be directly passed to the task-specific hidden layer of the classifier.

[0040] Figure 3-A depicts multiple classifiers, which are based on... Figure 2 The label tree depicted in the figure has a hierarchical arrangement that matches the hierarchical structure of leaf nodes and non-leaf nodes.

[0041] More specifically, Figure 3-A depicts a classifier 310 corresponding to the first level of the hierarchy, trained to determine the classification value assigned to label A for the aggregate embedding 308. A classifier 312 corresponding to the second level of the hierarchy is trained to determine classification values ​​for its inputs, particularly the binary classification value assigned to label A43. A classifier 314 corresponding to the second level of the hierarchy is trained to determine classification values ​​for its inputs, particularly the binary classification value assigned to label A44. A classifier 316 corresponding to the third level of the hierarchy is trained to determine classification values ​​for its inputs, particularly the binary classification value assigned to label A43B. A classifier 318 corresponding to the third level of the hierarchy is trained to determine classification values ​​for its inputs, particularly the binary classification value assigned to label A43C. A classifier 320 corresponding to the third level of the hierarchy is trained to determine classification values ​​for its inputs, particularly the binary classification value assigned to label A44B. The classifier 322, corresponding to the third level of the hierarchy, is trained to determine classification values ​​for its inputs, specifically the binary classification value assigned to label A44C.

[0042] During training, a shared layer can be used for both tasks. In this respect, a shared layer refers to an artificial neural network layer used to compute the classifiers for their respective outputs. For the classifiers for labels A43C and A43B, the shared layer includes layers for classifiers 308 (label A), 312 (label A43), 316 (label A43B), and 318 (label A43C), and a BERT layer, but not layers for classifiers 314 (label A44), 320 (label A44B), or 322 (label A44C).

[0043] If an instance is assigned a label, the binary classification in that example is true; otherwise, it is false. True indicates that the instance belongs to that label. False indicates that the instance does not belong to that label.

[0044] A softmax layer or softmax operation 324 can be provided for the classification values. As shown in the portion of the integration 300 depicted in Figure 3-A, seven softmax layers or softmax operations 324 can be used due to the seven labels across the hierarchy.

[0045] In this example, the classifier is arranged as a label-specific layer. In this example, determining whether an input can be assigned a label is a task. The classifier can typically be any type of classifier.

[0046] The task input utilizes the hierarchical relationships between labels and passes the learning from higher-level tasks to lower-level tasks. In this example, the input to classifier 310 is aggregate embedding 308.

[0047] The input to any classifier can include the output of a classifier at a higher level of the hierarchy. The input to such a classifier can include a concatenation of the aggregate embedding 308 with the output of a classifier at a higher level of the hierarchy. In this example, an optional means 326 is depicted for combining (e.g., concatenating) the aggregate embedding 308 with the corresponding output. According to one aspect, for second and lower levels, the aggregate embedding 308 or the CNN output can be combined, for example, concatenated with the hidden states of the parent layer of a dense layer passed as input to child layers.

[0048] In this example, the task is performed by at least one dense layer of the classifier. The output can be determined using a set of dense layers with a softmax function following the last dense layer.

[0049] The input to any classifier can be a vector. This vector can be passed through k-dense layers, each of size D.

[0050] The classifier is task-specific. The task-specific classifier in this example performs binary classification to predict whether a given instance (e.g., text or a sequence of words from a text or digit image) belongs to a specific category. This task-specific classifier in the example consists of a dense layer with a softmax output of 324 and a dimension of 2. This dense layer can have a rectified linear unit (ReLU) as the activation function and binary cross-entropy as the loss.

[0051] The integration 300 according to the second aspect is described below with reference to FIG3-B. Elements having the same function as those described above with reference to FIG3-A are labeled with the same reference numerals in FIG3-B.

[0052] The integration 300 according to the second aspect differs from the integration 300 according to the first aspect only in that the information of the label sequence t1, t2, t3, ... tn is aggregated into an aggregate embedding 308, such as a single vector. In contrast to the first aspect, the aggregate embedding 308 is not based on embeddings 307 for specific classification labels, such as CLS in BERT. Instead, the aggregation is determined by components 309 from the label embeddings 306-1, 306-2, ..., 306-n, where components 309 can be convolutional neural network layers, CNN layers, or aggregation functions such as averages, with or without attention mechanisms.

[0053] A set of convolutional layers with different kernel sizes can be applied to a tensor containing all the input labels. The convolutional layers in this example include kernels of sizes 2, 3, 4, and 5, and F filters per convolutional layer. In this example, the CNN layer is used as component 309 to create a single vector representing its entire input sequence.

[0054] The following is for reference. Figure 4 This describes a method for determining the computer implementation of knowledge graph 102. The following description assumes that there is a classifier for each leaf node or non-leaf node of the label tree.

[0055] This method uses a trained model. In this example, the model represents an ensemble of 300. This means that the model represents a neural network with multiple classifiers performing hierarchical classification. The model in this example is a neural network that includes layers and levels as described above. The linear layer in this example produces two logs for binary classification.

[0056] The method will be described using an exemplary hierarchy for the classifier and for the trained first classifier and the trained second classifier as examples.

[0057] A first classifier is trained to determine the first category. The first classifier is associated with the first node representing the first label in knowledge graph 102. A second classifier is trained to determine the second category. The second classifier is associated with the second node representing the second label in knowledge graph 102.

[0058] These hierarchical structures of classifiers can also include different classifiers trained to make different decisions. Classifiers at different levels within this hierarchy are assigned to different levels within the label hierarchy of the label tree. These classifiers can be used in the same manner as described for the first and second classifiers.

[0059] The links between parent and child nodes in knowledge graph 102 are represented by a hierarchical structure of classifiers. The classification or hidden state of the parent classifier can be used as input to the classifiers representing its children.

[0060] The first and second categories in this example are binary classifications for a specific label in the label tree.

[0061] According to this method, all classifiers can share the output of transformer 304. Different classifiers are trained in a multi-task setting described in more detail below. This means that the parameters of transformer 304 can also be trained in practice.

[0062] The method includes step 402, which uses a tokenizer 302 to segment text instances into a sequence of tokens.

[0063] Then proceed to step 404.

[0064] In step 404, for each tag, an embedding is determined using transformer 304, such as a transformer-based neural language model. Simultaneously, step 404 determines a single embedding. This single embedding can be an aggregate embedding 308, such as the output for CLS tags or component 309, which can be used to represent the entire sequence.

[0065] Then proceed to step 406.

[0066] In step 406, a first classification is determined for the embedding at the first classifier. For this purpose, in this example, the embedding is provided to the first classifier.

[0067] Then proceed to step 408.

[0068] In step 408, a second classification is determined at the second classifier. For this purpose, the embedding and / or the hidden state of the first classifier resulting from the embedding provided in step 406 can be provided as input to the second classifier.

[0069] Optionally, the embedding can be cascaded with the classification or the hidden state.

[0070] Then proceed to step 410.

[0071] In step 410, it is determined whether the first category satisfies the first condition. The first category can be either the instance belongs to the label or the instance does not belong to the label.

[0072] If the first category satisfies the first condition, such as indicating that the instance belongs to the label, then proceed to step 412. Otherwise, proceed to step 414.

[0073] In step 412, a first link is added between the first node representing the instance of knowledge graph 102 and the node representing the first category of knowledge graph 102.

[0074] Then proceed to step 414.

[0075] In step 414, it is determined whether the second category satisfies the second condition. The second category can be either the instance belongs to the label or the instance does not belong to the label.

[0076] If the second category satisfies the second condition, such as indicating that the instance belongs to the label, then proceed to step 416. Otherwise, proceed to step 418.

[0077] In step 416, a second link is added between the node representing the instance of knowledge graph 102 and the second node representing the second category of knowledge graph 102.

[0078] Then proceed to step 418. Step 418 is optional.

[0079] In step 418, when the second classification meets the conditions, a first link is determined from the node representing the instance to the first node representing the first classification. When the second classifier is at a lower level in the classifier hierarchy than the first classifier, the predictions integrated by the classifier are used for hierarchical prediction patterns or post-processing predictions are used to make the hierarchical predictions consistent. For example, if a related parent node has not yet been predicted but a child node has been predicted, the related parent node is added to the knowledge graph, or if a parent node has not yet been predicted, the child label is deleted.

[0080] In a multi-task setting, the main task is decomposed into a set of tasks corresponding to different classifiers. These classifiers output predictions separately for each task. Neural networks can have more... Figure 4 The hierarchical structure depicted in the graph has more levels. Therefore, it is easier to identify more complex knowledge graphs (102).

[0081] For instances of unseen digital images or text sequences, a set of labels is predicted in this manner using a multi-head neural network provided by the model.

[0082] This set of labels can optionally match the predicted hierarchical label set. To do this, parent labels can be added or child labels can be removed based on the predicted label set.

[0083] Node n can be added to the knowledge graph 102, which represents the original digital image or text sequence. A link labeled "belongsToCategory" can be added between n and any node L representing one of the predicted labels. These links may or may not include a parent category.

[0084] refer to Figure 5 The steps for training the model are described.

[0085] The model is provided in step 502.

[0086] Specifically, neural networks have a structure based on a label tree. A neural network can include multiple convolutional neural network layers, dropout layers, or dense layers.

[0087] In step 504, the tagged data points are provided.

[0088] In this example, the data points are assigned to a first category, a second category, or both. In this example, the data points include labels for both the first and second classifiers.

[0089] The training is performed using multiple labeled data points assigned to different or multiple categories.

[0090] Then proceed to step 506.

[0091] In step 506, a first classifier and / or a second classifier are trained based on the data points.

[0092] The model is trained to determine a first classification and / or a second classification. Therefore, a classifier, such as a convolutional network, at least one dense layer, or a portion of an artificial neural network, is trained for each label. There can be more than two levels in the hierarchy and classifier.

[0093] This means that different classifiers are trained to make different decisions. This combines the advantages of hierarchical classification algorithms with those of transformer-based pre-trained neural language models, which utilize training data optimally by training classifiers focused on different decisions, and which typically lead to state-of-the-art performance in text classification methods. Using hierarchical learning algorithms in multi-task settings makes this combination possible because it is not possible to store many transformer models in memory simultaneously. Since all classifiers share a single transformer model and different classifiers are trained in multi-task settings, computational feasibility is ensured even in settings with limited available computing power.

[0094] In one aspect, training the model includes determining a first loss from the output of a first classifier and a second loss from the output of a second classifier, and backpropagating the losses to train the weights in the first and second classifiers. In one example, the first and second classifiers do not belong to a shared layer. In this case, the determination of the first loss is independent of the output of the second classifier, and the parameters of the first classifier are updated based on the first loss and independently of the second loss. The determination of the second loss is independent of the output of the first classifier, and the parameters of the second classifier are updated based on the second loss and independently of the first loss. In another example, the first and second classifiers belong to a shared layer. In this example, the shared layer is updated based on the first and second losses. The parameters of the first and second classifiers are updated, for example, based on the sum of the first and second losses. The first or second loss can be determined as the sum of the norms of the difference between the softmax output 324 of the input defined in the data points and the reference defined for that input in the data points. The input can be an aggregate embedding 308, such as an embedding of CLS tags or an embedding of information aggregated in the training instance or the sequence of tags t1, ..., tn of the training instance.

[0095] On the other hand, training the model includes determining a loss from the output of the first classifier and backpropagating that loss to train weights in the first classifier but not in the second classifier when the data points are labeled for a first classification but not for a second classification.

[0096] On the other hand, training the model includes determining a loss from the output of the second classifier and backpropagating that loss to train the weights in the second classifier instead of the first classifier when the data points are labeled for a second classification rather than for a first classification.

[0097] The training may include multiple periods.

[0098] The training steps can be performed before using the model described above. In this case, step 402 is performed after the training.

[0099] The method may include the step of determining a classification for input to a model trained in this manner. The classification may be performed using a classifier located at a position in the classifier hierarchy of the model. The method may also include assigning the classification to a label corresponding to the position in the hierarchy of a label tree.< / cls> < / cls>

Claims

1. A method for determining a computer-based implementation of a knowledge graph, characterized in that: Determine the embedding of the tag sequence of the (404) instance; The first classifier of the embedding is determined at the first classifier (406), which is an ensemble of multiple classifiers arranged according to a hierarchical structure, including the first classifier. Determine (410) whether the first category satisfies the first condition; When the first category satisfies the first condition, add (412) a first link between the first node representing the instance in the knowledge graph and the node representing the first category in the knowledge graph; otherwise, do not add the first link. The embedding and the hidden state of the first classifier resulting from the embedding are provided as input to the second classifier, which is at a lower level than the first classifier in the hierarchy. The marker sequence is formed by segmenting instances of text.

2. The method according to claim 1, characterized in that, At the second classifier, determine (408) the second category; determine (414) whether the second category satisfies the second condition; when the second category satisfies the second condition, add (416) a second link between the node representing the instance in the knowledge graph and the second node representing the second category in the knowledge graph; otherwise, do not add the second link.

3. The method according to claim 1, characterized in that, When the second category meets the condition, determine (418) the first link from the node representing the instance to the first node representing the first category.

4. The method according to any one of claims 1-3, characterized in that, Determine the tag sequence of the instance described in (402).

5. The method according to any one of claims 1-3, characterized in that, The examples include digital text data.

6. The method according to any one of claims 1-3, characterized in that, The first classification and / or the second classification is a binary classification.

7. The method according to any one of claims 1-3, characterized in that, Provide (502) nodes representing the tag tree to the knowledge graph, and add (506) multiple links to the nodes representing the tags of the instance to the knowledge graph.

8. The method according to claim 7, characterized in that, The first classification and / or the second classification are used to determine whether the instance belongs to a category represented by a node in the label tree.

9. The method according to any one of claims 1-3, characterized in that, The first category is assigned (502) a different label than the second category.

10. The method according to claim 2, characterized in that, Provide (504) data points including labels for the first classification and / or labels for the second classification, and train (506) the first classifier and / or the second classifier based on the data points.

11. The method according to claim 2, characterized in that, Provide (502) a neural network, and train (506) the neural network to determine the first classification or the second classification.

12. The method according to claim 11, characterized in that, Training (506) the neural network includes determining a loss from the output of the first classifier and a loss from the output of the second classifier, and backpropagating the losses to train the weights in the first classifier and the second classifier based on the two losses, or training the first classifier based on the first loss and independent of the second loss, and training the second classifier based on the second loss and independent of the first loss.

13. The method according to claim 11, characterized in that, The classifier located at a position in the classifier hierarchy of the neural network determines the classification for the input of the neural network trained in this manner, and assigns the classification to the label corresponding to the position in the hierarchy of the label tree.

14. A device (100) for determining a knowledge graph (102), characterized in that, The device (100) is configured to perform the steps of the method according to any one of claims 1-13.

15. A computer program product, characterized in that, The computer program product includes computer-readable instructions that, when executed by a computer, cause the computer to perform the steps of the method according to any one of claims 1 to 13.

Images