Apparatus for automatically generating a knowledge graph
By using multilingual models and gradient inversion training, the problem of monolingual models being unable to recognize cross-lingual tense expressions was solved, thus improving the accuracy of cross-lingual tense expression recognition and knowledge graph filling.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ROBERT BOSCH GMBH
- Filing Date
- 2021-04-15
- Publication Date
- 2026-07-03
AI Technical Summary
Existing monolingual models cannot effectively identify and transfer cross-lingual temporal representations, leading to errors when populating knowledge graphs with multilingual text.
Using a multilingual model and gradient inversion training method, a multilingual word embedding space is generated through adversarial training. This space is used to identify and label tense expressions in multilingual texts and apply them to fill knowledge graphs.
It enables cross-linguistic tense expression recognition and transfer, improving the accuracy of knowledge graph filling and the effectiveness of multilingual text analysis.
Smart Images
Figure CN113535971B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a device for automatically analyzing multilingual text, a training system for training the device, a device for automatically generating a knowledge graph, and a machine-readable storage medium. Background Technology
[0002] An ontology, also known as a knowledge graph, is a structured representation of entities and the relationships between them. Ontologies are used to formally exchange knowledge between computer-implemented applications.
[0003] The identification of temporal representations plays a crucial role in extracting information from text to populate knowledge graphs: for example, certain relationships between entities are valid only for a limited time (e.g., a person's place of residence or employer). So-called "temporal tagging" aims to automatically identify temporal representations in text. To date, all work on this topic has used monolingual models, i.e., language-specific models, which can only be applied to text in their specific language and not to text in other languages.
[0004] In their 2015 arXiv preprint arXiv:1505.07818v1, "Domain-Adversarial Training of Neural Networks," Yaroslav Ganin et al. disclosed a method for training neural networks based on labeled data from a source domain and unlabeled data from a target domain (i.e., without requiring labeled data from the target domain). Further training facilitates the emergence of features that are (i) discriminative for the primary learning task in the source domain and (ii) indistinguishable in terms of domain-to-domain transfer. The method disclosed in this paper is also known as gradient reversal training. Summary of the Invention
[0005] Compared to the prior art, the subject matter of independent claim 1 has the following advantages: the subject matter can be applied to texts in multiple languages. Furthermore, cross-language transferability is also possible.
[0006] Advantageous extensions are the subject of dependent claims, while other aspects are the subject of co-claims.
[0007] For example, if a model learns from English text that "today" is a time expression, but the word "heute" never appears in the German training text, then the monolingual model cannot interpret the word "heute" as a time expression. However, a multilingual model can learn from other sources (e.g., through multilingual embeddings) that "today" and "heute" have similar meanings and thus correctly identify the word "heute" as a time expression.
[0008] Therefore, the solution to this problem can be divided into two sub-steps:
[0009] 1. Training and applying multilingual models based on data from multiple languages.
[0010] 2. Alignment of number representations, also known as word embeddings, allows for the identification of semantic similarities across languages.
[0011] Therefore, in a first aspect, the present invention relates to an apparatus for automatically analyzing multilingual text in multiple languages among a plurality of pre-given languages, the apparatus comprising an embedder and a temporal tagger, wherein the embedder is configured to assign numerical representations (i.e., word representations) to text components (e.g., words or parts of words) in the multilingual text, wherein the temporal tagger is configured to identify and tag tense expressions in the multilingual text according to the assigned embeddings, wherein the embedder is configured to assign the numerical representations of the tense expressions as much as possible such that it is impossible to determine which language the text component to which it belongs was written based on the numerical representations.
[0012] It is possible to solve the above two problems by using a unique neural model, which is trained on data in different languages and generates a multilingual vector space for word embeddings through adversarial training (especially gradient inversion training), in which a discriminator (a second neural network) cannot distinguish which language a particular word comes from.
[0013] This invention can be used to automatically extract tense expressions from text. This is an important sub-aspect in the context of knowledge graph population, because certain facts are only valid for a specific period of time. This validity period must be marked in the knowledge graph so that errors are not generated when the graph is applied.
[0014] While the method known from Ganin et al. (2015) can in principle be applied to all expressions appearing in a sentence, it is actually done in a more advantageous implementation: instead of first extracting tense expressions and then performing linguistic identification with gradient inversion, both are applied in parallel; that is, linguistic identification performed with gradient inversion is applied to all expressions, regardless of whether those expressions are tense. It is now recognized that using this method to identify tense expressions is advantageous because tense expressions have similar structures across a large number of languages.
[0015] Conversely, if, for example, one wants to extract grammatical structures, such as the fourth case accusative, then such cross-language transfer may not be so purposeful, because the fourth case accusative in one language may have a completely different grammatical function in another language.
[0016] Therefore, this invention enables the use of multilingual models (instead of monolingual models) for tense markers. This is more advantageous than using rule-based models, which do not transfer well to new languages.
[0017] Experiments have shown that the proposed model performs much better than known comparative models in the prior art for transferring to unknown languages.
[0018] Another advantage of this invention is that it utilizes gradient inversion to improve the multilingual embedding space (this method is known from the prior art to be available only at higher network levels, and the multilingual embedding in the input is merely a means to an end, but not the focus of improvement).
[0019] This aspect of the invention can be used to automatically extract temporal expressions from text. This is an important sub-aspect in the case of populating knowledge graphs, because certain facts, i.e., entities and / or relationships, are only valid for a specific period of time. This validity period can be marked in the knowledge graph so that errors are not generated when the graph is applied.
[0020] Therefore, a (particularly neural) multilingual model is proposed that can automatically extract tense expressions from texts in different languages so that these tense expressions can be inserted into a knowledge graph as additional information about the facts.
[0021] Furthermore, it is proposed to use adversarial training (especially gradient inversion training) to train the model in order to generate a multilingual word embedding space in which languages cannot be distinguished from each other.
[0022] In adversarial training, it can be stipulated that the objective function used to train the tense tokenizer alternates with the objective function used to train the discriminator (abwechseln), and that the embedding vector space is updated, for example, by gradient inversion, so that the discriminator is as indistinguishable from the language as possible. That is, it is recognized that this leads to semantically similar tense expressions from different languages obtaining similar word representations.
[0023] As a result, the model can be applied to text in any language, without explicitly specifying which language it is. Furthermore, the model can also be applied to languages not derived from a set of languages on which it was trained. The only prerequisite is that the language exists in a multilingual embedding. This is where the model's transfer performance compared to monolingual or rule-based systems lies.
[0024] In other words, during the training phase, it can be assumed that the training data has characteristic tense expressions from different languages. The embedding unit, particularly the neural model (e.g., the advanced BERT model), is trained or (in the case of BERT) adapted based on this training data. In the case of BERT, this can be done through fine-tuning: BERT is typically pre-trained based on language modeling tasks and can be adapted to the target task through fine-tuning.
[0025] For this purpose, multilingual embeddings can be used, i.e., a unique vector space for embeddings from all languages. Specifically, the following steps can be performed: First, the training data can be divided into batches, where each batch is ensured to contain examples from multiple, and in particular, all, languages.
[0026] The neural model is trained using adversarial training with training batches. Here:
[0027] a) On the one hand, this trains the output of the temporal marker so that the temporal marker can identify as many temporal expressions as possible in the training data;
[0028] b) On the other hand, the discriminator is trained such that it attempts to determine the language of the training example given the multilingual embeddings of that batch. An embedding vector space is generated through gradient inversion training, in which the discriminator cannot (or has difficulty) distinguishing the individual languages from one another.
[0029] To apply this model, it can be assumed that multilingual texts are provided, that is, single or multiple texts from different languages.
[0030] The trained tense marker can now be applied to these texts. It is not necessary to provide the tense marker with information about the language from which the text originates, because the word representation space is multilingual and the tense marker has been trained to recognize temporal expressions in a language-independent manner.
[0031] The identified temporal expressions can then be used in a known manner to automatically build or populate knowledge graphs. Attached Figure Description
[0032] Embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. In the drawings:
[0033] Figure 1 The structure of an embodiment of a device for analyzing multilingual text is illustrated schematically;
[0034] Figure 2 The structure of an embodiment of the training system is illustrated schematically. Detailed Implementation
[0035] Figure 1 The structure of a device (100) for analyzing multilingual text is schematically shown. Text (10), which may include one or more documents and may be written in multiple languages, is first fed to an analyzer (25), which extracts text components (10a, 10b) from the text (10). Alternatively, the text components (10a, 10b) may be fed directly to the device (100) for analyzing multilingual text. These text components are then fed to an embedder (25), in this embodiment provided by an artificial neural network, such as a BERT model, which determines the corresponding digital representation (12a, 12b) for each text component (10a, 10b).
[0036] These numerical representations (12a, 12b) are fed to a temporal marker (30), which is also provided by a neural network and determines a flag (13a, 13b) for each numerical representation (12a, 12b), indicating whether the text component (10a, 10b) to which the corresponding numerical representation (12a, 12b) belongs is a tense expression.
[0037] The flags (13a, 13b) are fed together with the text components (10a, 10b) to the merger (32), which merges the flags (13a, 13b) into their respective text components (10a, 10b) so as to determine the tense-marked text components (11a, 11b).
[0038] These temporally tagged text components can be fed to an optional knowledge graph builder (40), which provides an existing knowledge graph (41) and identifies entities and relationships between these entities in the temporally tagged text components (11a, 11b), assigns the identified temporal expressions to the entities and / or the relationships, and considers the identified temporal expressions as representations of the assigned entities and / or relationships when generating the knowledge graph.
[0039] The device (100) for analyzing multilingual text can be, for example, a computer system, wherein the aforementioned components (20, 25, 30, 32, 40) can be implemented as computer programs stored on a machine-readable storage medium (101).
[0040] Figure 2 The structure of a training system (200) for training a device (100) for analyzing multilingual text is illustrated schematically. Figure 1 As shown, the text components (10a, 10b) are fed to an embedder (25), which (25a) thereby determines the numerical representations (12a, 12b), as... Figure 1 As described, the tense markers (13a, 13b) can be determined from the numerical representations by means of a tense marker (30). The numerical representations (12a, 12b) are fed to a discriminator (50), which attempts to determine, based on these numerical representations (12a, 12b), which to which language the associated text components (10a, 10b) belong. In this embodiment, the discriminator (50) is provided by an artificial neural network.
[0041] The estimated corresponding language is transmitted to the evaluator (60) in a manner encoded in the estimated linguistic markers (14a, 14b), as are the actual linguistic markers (15a, 15b), which serve as ground truths describing the actual language of the language components (10a, 10b). The evaluator (60) then uses a cost function to determine how well the estimated linguistic markers (14a, 14b) and the actual linguistic markers (15a, 15b) are consistent, and modifies the parameters of the discriminator (50) to make the consistency as good as possible, and modifies the parameters of the embedder (25) to make the consistency as poor as possible. Mathematically, this is achieved by having the sign bit of the gradient of the embedder (25) parameters opposite to the sign bit of the gradient of the discriminator (50) during backpropagation of gradients.
[0042] The temporal marker (30) can be trained as part of the training system (200) or outside of the training system (200). Preferably, the training data is divided into batches, wherein each batch is ensured to contain examples from all languages. The temporal marker (30) is trained using the training data such that the temporal marker identifies as many temporal expressions as possible in the training data, and the same training data is used to train the embedder (25) and the discriminator (50).
[0043] The training system (200) may be, for example, a computer system, wherein the aforementioned components (50, 60) may be implemented as computer programs stored on a machine-readable storage medium (102).
[0044] It should be understood that these methods can be implemented entirely in software, as described. They can also be implemented in hardware or a hybrid of software and hardware.
Claims
1. A training system (200) for training a device for automatically analyzing multilingual text, the training system comprising: A device for automatically analyzing multilingual text, the device comprising: Embedder (25), comprising a machine learning system, configured to assign numerical representations (12a, 12b) to text components (10a, 10b) in the multilingual text (10), and A tense marker (30) is configured to identify and mark tense expressions in the multilingual text (10) according to the assigned numerical representations (12a, 12b); A discriminator (50) is configured to receive the digital representations (12a, 12b) from the embedder (25) and attempt to determine, based on the digital representations (13a, 13b), which language the text components (10a, 10b) belonging to the discriminator are of; An evaluator (60) is configured to receive from the discriminator (50) linguistic markers (14a, 14b) indicating the estimated corresponding language and linguistic markers (15a, 15b) indicating the actual language, wherein the evaluator (60) is further configured to modify the parameters of the machine learning system of the embedder (25) such that the embedder (25) is configured to distribute the numerical representations of the tense expressions (12a, 12b) as much as possible such that it is impossible to determine, based on the numerical representations (12a, 12b), which to which the text component (10a, 10b) is written; The training system (200) is configured to perform adversarial training on the discriminator (50) and the embedder (25).
2. The training system (200) according to claim 1, wherein the training system is configured to train the discriminator (50) and the embedder (25) using a gradient inversion method.
3. The training system (200) according to claim 1, wherein the training system is configured to divide the training data used in training the discriminator (50) and the embedder (25) into batches, such that each batch contains training examples from a plurality of languages selected from a plurality of languages that can be pre-given.
4. The training system (200) according to claim 3, wherein the training system is configured to divide the training data used in training the discriminator (50) and the embedder (25) into batches such that each batch contains training examples from each of a plurality of languages that can be pre-given.
5. The training system (200) according to any one of claims 1 to 4, wherein the training system is configured to determine, by means of the tense marker (30), whether the associated text component (10a, 10b) is a tense expression, and to perform training of a discriminator (50) and / or an embedder (25) for the expression wherein the associated text component (10a, 10b) has been determined to be a tense expression.
6. The training system (200) according to any one of claims 1 to 4, wherein the training system is configured to further train a tense marker (30) according to the following training objective: the tense marker is able to determine as well as possible whether the text module (10a, 10b) to which it belongs is a tense expression based on the numerical representation (12a, 12b).
7. The training system (200) according to claim 6, wherein the training system is configured to train the temporal marker (30), the embedder (25) and the discriminator (50) based on the same training examples.
8. The training system (200) according to any one of claims 1 to 4, wherein, The embedder (25) is configured to map all tense expressions of all languages to a unique vector space.
9. The training system (200) according to any one of claims 1 to 4, wherein, The machine learning system includes a neural network, which is a BERT model.
10. The training system (200) according to any one of claims 1 to 4, wherein, The discriminator (50) includes a second machine learning system.
11. The training system (200) according to claim 10, wherein, The second machine learning system includes a second neural network.
12. The training system (200) according to any one of claims 1 to 4, wherein, The discriminator (50) has been trained to determine, as accurately as possible, which language the tense expression belongs to based on the numerical representation (12a, 12b) of the tense expression.
13. The training system (200) according to any one of claims 1 to 4, wherein, The temporal marker (30) includes a third machine learning system.
14. The training system (200) according to claim 13, wherein, The third machine learning system includes a third neural network.
15. The training system (200) according to claim 13, wherein, The tense marker (30) has been trained to determine, as accurately as possible, whether the text module (10a, 10b) to which it belongs is a tense expression based on the numerical representation (12a, 12b).
16. The training system (200) according to claim 13, wherein, The discriminator (50), the embedder (25), and the temporal marker (30) have been jointly trained adversarially.