Extracting open information from low-resource languages

Abstract

To provide a method for extracting a machine-readable data structure from a low resource language input text that is not structured.SOLUTION: A method includes: a step 810 for acquiring a corpus having a high resource language data structure; a step 820 of filtering the corpus having the high resource language data structure; a step 830 for acquiring entity types as each of the filtered data structures; a step 840 of substituting each of the entities with the entity of the same type, and thereby substituting the types; a step 850 of performing data structure code switching so as to generate a code switched sentence; a step 860 for generating a complemented data structure corpus; a step 870 for training a multi-head self attention transformer model; and a step 880 of giving a data structure that is not structured so as to generate a low resource language data structure.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] Cross-reference to Related Applications Priority is claimed to U.S. Patent Application No. 63 / 165,755, filed Mar. 25, 2021, the entire disclosure of which is hereby incorporated by reference herein.

[0002] The present invention relates to artificial intelligence and machine learning, and more particularly, to a method, system, and computer-readable medium for extracting information in the form of machine-readable data structures from low-resource languages using transfer learning from one or more high-resource languages.

Background Art

[0003] The technical problem of extracting structured data from unstructured text in the form of (subject, verb, object) triples has become important in the area of intelligent learning and machine learning applications due to the wide spread of applications such as knowledge graph creation, document summarization, link prediction, and question answering. One example of such an application is to extract news events and facts from news articles in the form of a knowledge graph and infer additional information using link prediction. Existing open information extraction (OpenIE) methods mainly focus on the English language using supervised and unsupervised methods. Ro, Youngbin et al., “Multi 2Several existing OpenIE methods are described in "OIE: Multilingual Open Information Extraction based on Multi-Head Attention with BERT", arXiv preprint arXiv:2009.08128 (October 7, 2020), and in "OpenIE6: Iterative Grid Labeling and Coordination Analysis for Open Information Extraction" by Kolluru, Keshav et al., arXiv preprint arXiv:2010.03147 (October 7, 2020). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Ro, Youngbin et al., “Multi2OIE: Multilingual Open Information Extraction based on Multi-Head Attention with BERT,” arXiv preprint arXiv:2009.08128 (October 7, 2020) [Non-Patent Document 2] Kolluru, Keshav et al., “OpenIE6: Iterative Grid Labeling and Coordination Analysis for Open Information Extraction,” arXiv preprint arXiv:2010. 03147 (October 7, 2020) [Overview of the project] [Means for solving the problem]

[0005] According to one embodiment, a method is provided for extracting machine-readable data structures from unstructured, low-resource language input text. The method includes the steps of: obtaining a corpus of high-resource language data structures; filtering the corpus of high-resource language data structures to obtain a filtered corpus of high-resource language data structures; obtaining the entity type for each entity of each filtered high-resource language data structure; performing type substitution for each obtained entity by replacing each entity with an entity of the same type to generate a type-substituted data structure; and replacing each entity with an equivalent corresponding low-resource language data structure entity to generate a code-switched sentence. The method further includes generating an augmented data structure corpus, which involves combining type-substituted data structures and code-switched sentences with a filtered high-resource language data structure corpus; training a multi-head self-attention transformer model using the augmented data structure corpus with dynamic iterative tagging and gradient descent algorithms; and feeding unstructured low-resource language input text to the trained multi-head self-attention transformer model to extract machine-readable data structures.

[0006] Embodiments of the present invention will be described in more detail below with reference to illustrative drawings. The present invention is not limited to illustrative embodiments. All features described and / or illustrated herein may be used individually or in different combinations in embodiments of the present invention. Features and advantages of various embodiments of the present invention will become apparent by reading the following detailed description with reference to the accompanying illustrations below. [Brief explanation of the drawing]

[0007] [Figure 1] This figure schematically illustrates a method and system for extracting triples from an input sentence according to one embodiment of the present invention. [Figure 2] This figure schematically illustrates a method and system for selecting English triples having unambiguated arguments, according to one embodiment of the present invention. [Figure 3] This figure shows an example of a triple that completes both the subject and the object. [Figure 4] This figure schematically illustrates a method and system for generating complementary data from high-resource language triples according to one embodiment of the present invention. [Figure 5] This figure schematically illustrates a combined method and system for generating complementary data and extracting triples using the complementary data, according to one embodiment of the present invention. [Figure 6] This diagram schematically illustrates an exemplary application of one embodiment of the present invention for news articles. [Figure 7] This flowchart shows the operation of a multi-head self-aware transformer model across the entire system according to one embodiment of the present invention. [Figure 8] This figure shows a flowchart of a data method according to one embodiment of the present invention for extracting machine-readable data structures from unstructured, low-resource language input text. [Modes for carrying out the invention]

[0008] Embodiments of the present invention provide methods, systems, and computer-readable media for extracting information from unstructured, low-resource language texts (e.g., Japanese, Korean, Hindi, etc.) in the form of machine-readable data structures, such as (subject, verb, object) triples. The methods, systems, and computer-readable media described herein employ transfer learning from high-resource languages ​​(e.g., English) by utilizing data completion and iterative head prediction techniques. In contrast to existing methods, the techniques described herein are the first to leverage code-switched data along with type-switching and complementary concatenation to enable language transfer from high-resource languages ​​to low-resource languages. Code-switched data represents unstructured text and / or structured data containing entities from multiple languages. For example, a code-switched sentence is a sentence containing entity words from multiple languages; for example, “Москва is the capital of Россия” and “Wien is the capital of Osterreich” are examples of code-switched sentences. Type-switched data represents a sentence in which entities have been switched with other entities of the same type. For example, “London is the capital of Russia” is a type-switched sentence where the entity “Moscow” is switched with a different entity of the same type (e.g., type “city”) (i.e., “London”). Similarly, (London, is capital of, Russia) is a type-switched data structure where the entity “Moscow” is switched with the entity “London”. Complementary concatenations show sentences where additional entities use concatenations.For example, you could start with unstructured text like "Rhine flows through Karlsruhe" and add additional entities to it to get the completed concatenation "Rhine flows through Karlsruhe and Mannheim and Frankfurt and Berlin". The original unstructured text is then complemented by entities belonging to the same type (for example, type "city") using concatenation.

[0009] Due to a lack of training data and linguistic expertise, OpenIE (i.e., Open Information Extraction) methods have not been developed for low-resource languages. However, the techniques described herein make it possible to transfer linguistic knowledge to low-resource languages ​​by leveraging data from high-resource languages ​​such as English. Specifically, the methods described herein accurately extract triples from low-resource language text using the data completion and iterative head tagging procedures of the present invention, without requiring the training of a model with low-resource language training data. Since training data in low-resource languages ​​is usually unavailable, the ability to extract information from unstructured low-resource language text in the form of machine-readable data structures, such as (subject, verb, object) triples or (subject, predicate, object) triples, without the need for training data in low-resource languages, represents a significant technological advancement.

[0010] According to a first aspect, a method is provided for extracting machine-readable data structures from unstructured, low-resource language input text. The method includes the steps of obtaining a corpus of high-resource language data structures, filtering the high-resource language data structures to obtain a filtered corpus of high-resource language data structures, and obtaining the entity type for each entity in each filtered high-resource language data structure. The method further includes the step of generating a completed data structure corpus, which includes the steps of performing type substitution for each obtained entity by replacing each entity with an entity of the same type to generate type-substituted data structures, replacing each entity with an equivalent low-resource language data structure to generate code-switched sentences, and combining the type-substituted data structures and code-switched sentences with the filtered high-resource language data structure corpus. In addition, the method includes the steps of training a multi-head self-attention transformer model using the completed data structure corpus with dynamic iterative tagging and gradient descent algorithms, and feeding unstructured, low-resource language input text to the trained multi-head self-attention transformer model to extract machine-readable data structures.

[0011] An implementation of a method for extracting machine-readable data structures from unstructured, low-resource language input text may include the step of obtaining a corpus of high-resource language (subject, verb, object) triples, and then filtering out specific triples to obtain a filtered corpus of triples. The method further includes the step of obtaining the entity type for each entity in each triple of the filtered triple corpus. The entity type can be any defined taxonomic rank or classification level to which the entity (i.e., language unit) belongs, e.g., class / subclass / grouping / etc. For example, for the language unit "Amazon", it may be possible to obtain the types "river", "body of fresh water", "flowing body of water", and "body of water".

[0012] Subsequently, the method includes a step of performing a type substitution procedure, thereby replacing the triple entities in the filtered triple corpus with entities of the same type to obtain type-substituted triples. For example, in a (subject, predicate, object) triple that constitutes argument pairs (i.e., subject and object) and inter-argument relationships (i.e., predicates), the arguments are replaced with other arguments of the same type to obtain type-substituted (subject, predicate, object) triples. For example, the type substitution procedure could include generating type-substituted triples (Alps, are located in, Africa), (Himalayas, are located in, North America), (Andes, are located in, Europe) and type-substituted triples (Mexico City, is capital of, United Kingdom), (Moscow, is capital of, China), and (London, is capital of, Germany) from the high-resource language triples (Alps, are located in, Europe) and (London, is capital of, United Kingdom). In each case, the type substitution procedure involves replacing an entity of the extracted triple (e.g., "Alps") with another entity of the same type (e.g., "Himalayas" or "Andes"), where the entity type (e.g., "mountain ranges") is used. During the training phase of the transformer model, which will be discussed later, it is irrelevant whether the relationships between terms in a particular triple are true or false. During the subsequent operational phase, the transformer model's ability to extract coherent triples will improve by adding valid relationships, regardless of whether they are true or false.

[0013] This method additionally includes the step of replacing entities with their equivalent low-resource language names in order to obtain code-switched sentences. For example, starting with the triples (Vienna, is capital of, Austria) and (Moscow, is capital of, Russia), the code-switched sentences "Москва is the capital of Россия" and "Wien is the capital of Osterreich" can be obtained. Similarly, the code-switched sentence "Wien is the capital of Deutschland" can be obtained from the type-switched data structure (Vienna, is capital of, Germany).

[0014] Once type-substituted triples and code-switched sentences are obtained, they are added to a filtered corpus to provide a supplemented data structure corpus. Notably, the supplemented data structure corpus includes both high-resource language data structures, e.g., high-resource language (subject, verb, object) triples, and low-resource language data structures, e.g., low-resource language (subject, verb, object) triples. After the supplemented data structure corpus is obtained, the method according to the first aspect further includes the step of training a multi-head self-attentional transformer model with dynamic iterative tagging using a gradient descent algorithm, e.g., a stochastic gradient descent algorithm, by using the supplemented data structure corpus. In short, to train the multi-head self-attentional transformer model, a high-resource language data structure corpus is obtained, type-switching is performed to provide type-switched data structures, and then code-switched unstructured text is created from the type-switched data structures. The code-switched unstructured text is then provided as training data, along with the underlying sentences (provenance sentences) corresponding to the high-resource language triples. To evaluate the output of the transformer model during training, both type-switched data structures and the original high-resource language data structures are used.

[0015] Finally, after training, the method includes the steps of splitting the unstructured input text, which may include high-resource language text, low-resource language text, or a combination thereof, into sentences in order to extract machine-readable data structures from the unstructured input text, such as (subject, verb, object) triples, and inputting those sentences into a trained model.

[0016] Therefore, a method for extracting low-resource language (subject, predicate, object) triples from either unstructured input text in a high-resource language or unstructured input text in a low-resource language includes the steps of complementing high-resource language data, training a multi-head self-attention transformer model using the complemented high-resource language data, and inputting the unstructured input text into the trained multi-head self-attention transformer model.

[0017] High-resource language triples can be obtained from at least one open information exchange (OIE) system, such as a corpora that is publicly available. The entity types obtained for each of the filtered high-resource language data structures can also be obtained from such an open information exchange system.

[0018] The steps of training a multi-head self-attention transformer model include inputting a complemented data structure corpus containing high-resource language triples of the complemented data structure corpus into the multi-head self-attention transformer model, and dynamically and iteratively tagging tokens using the step of tagging the heads multiple times. Following the tagging step, the weights of the multi-head self-attention transformer model are updated using, for example, a gradient descent algorithm. The input to the transformer during training is a sentence from a complemented dataset (i.e., code-switched unstructured text and the underlying sentences corresponding to high-resource language triples) along with tags, and the output during the training phase is a trained model obtained using gradient descent optimization. The weights of the multi-head self-attention transformer model are updated using, for example, the gradient descent algorithm and cross-entropy loss during training, which can be performed multiple times for the entire training dataset until, for example, the obtained scores regarding the validation dataset, such as the F1 score, stop improving.

[0019] The steps of providing unstructured high-resource language text and / or low-resource language text to a trained multi-head self-attention transformer model include separating the high-resource language text and / or low-resource language text into individual high-resource language sentences and / or low-resource language sentences, and inputting the individual sentences into the trained multi-head self-attention transformer model to obtain a machine-readable data structure, such as (subject, verb, object) triples.

[0020] High-resource language triples in a high-resource language data structure corpus have fields for subjects, verbs, and objects, each corresponding to a specific entity type. The entity type for each field in each triple can be obtained from the OIE system from which the triple is acquired. Open information exchange systems include internet databases, datasets, and knowledge bases, which contain data in the form of open data, linked data, or linked open data.

[0021] According to one embodiment of the method, a high-resource language data structure includes subject, verb, and object triples, and the entity type for each entity of the filtered high-resource language data structure is obtained from at least one open information exchange (OIE) system, and the supplemented data structure corpus includes high-resource language data structures and low-resource language data structures. The entity type for the high-resource language data structure may include, for example, subject and object classifiers. The at least one OIE system may include, for example, open data, linked data, or a database of linked open data.

[0022] According to one embodiment of the method, the step of training a multi-head self-attention transformer model includes the step of updating the weights of the multi-head self-attention transformer model using a gradient descent algorithm and cross-entropy loss, and repeating the updating step multiple times until the score obtained on the validation dataset no longer improves. The step of updating the transformer model weights may include a step of dynamic iterative tagging, which may include tagging the first token head of a high-resource language triple (subject token head, verb token head, or object token head), then tagging another token head among the subject, verb, and object token heads, and then tagging the remaining token head among the subject, verb, or object token heads. The order of dynamic head token tagging of a high-resource language triple is performed dynamically using entropy. Entropy is measured using a softmax operation to measure the predictive accuracy confidence of head tokens from minimum to maximum, and the determination of the order of head token tagging is performed by aggregating only the overhead tokens that result in positive head token tags of the high-resource language triple. After dynamic head token tagging is performed using a first dynamic prediction order, at least one additional dynamic head token tagging of high-resource language triples is performed using another dynamic prediction order, and triples predicted by multiple dynamic prediction orders are marked as high-confidence triples. The dynamic head token tagging order can be selected randomly.

[0023] According to one embodiment of the method, the step of feeding unstructured low-resource language input text to a trained multi-head self-attention transformer model includes the steps of separating the unstructured low-resource language input text into individual low-resource language sentences and inputting the individual low-resource language sentences into the trained multi-head self-attention transformer model to obtain subject, verb, and object triples.

[0024] According to one embodiment of the method, training a multi-head self-attention model generates a monolingual and / or multilingual knowledge graph linking entities. According to one embodiment of the method, unstructured low-resource language input text is a code-mixed sentence, and the code-mixed sentence includes a sentence in which languages ​​are mixed. According to one embodiment of the method, the unstructured low-resource language input text includes text in a first low-resource language, and the extracted machine-readable data structure includes a data structure for the first low-resource language.

[0025] In a second aspect, the disclosure describes a non-temporary computer-readable medium storing instructions for performing a method for extracting machine-readable data structures from unstructured, low-resource language input text. The method includes the steps of: obtaining a corpus of high-resource language data structures; filtering the high-resource language data structures to obtain a filtered corpus of high-resource language data structures; and obtaining an entity type for each entity in each filtered high-resource language data structure. The method further includes the step of generating a complementary data structure corpus, which includes the steps of: performing type substitution for each obtained entity by replacing each entity with an entity of the same type to generate type-substituted data structures; replacing each entity with an equivalent low-resource language data structure to generate code-switched sentences; and combining the type-substituted data structures and code-switched sentences with the filtered high-resource language data structure corpus. In addition, the method includes the steps of training a multi-head self-attention transformer model using a complementary data structure corpus with dynamic iterative tagging and gradient descent algorithms, and feeding unstructured, low-resource language input text to the trained multi-head self-attention transformer model to extract machine-readable data structures.

[0026] According to a third aspect, the Disclosure provides a system for extracting machine-readable data structures from unstructured, low-resource language input text. The system includes one or more processors configured to obtain a corpus of high-resource language data structures, to filter high-resource language data structures to obtain a filtered corpus of high-resource language data structures, and to obtain an entity type for each entity in each filtered high-resource language data structure. The one or more processors are further configured to generate a complementary data structure corpus, which includes performing type substitution for each obtained entity by replacing each entity with an entity of the same type to generate type-substituted data structures, replacing each entity with an equivalent low-resource language data structure to generate code-switched sentences, and combining the type-substituted data structures and code-switched sentences with the filtered high-resource language data structure corpus. In addition, one or more processors are configured to train a multi-head self-attention transformer model using a complementary data structure corpus with dynamic iterative tagging and gradient descent algorithms, and to feed unstructured, low-resource language input text to the trained multi-head self-attention transformer model in order to extract machine-readable data structures.

[0027] Figure 1 schematically illustrates a method and system for extracting triples using data supplemented from a data supplementation method, as further described later, according to embodiments of the present invention. The system is based on an OpenIE extraction model having a self-attention-based neural sequence model, such as a transformer model, which includes three token classification heads: a subject head, a predicate head, and an object head. The token classification heads are dense (linear) layers, followed by the application of a softmax operation to all tokens in the input sentence.

[0028] The input sentence is fed into a self-attention-based neural sequence model, such as a transformer model, which transforms the input sentence into a series of embeddings, each embedding corresponding to a word in the sentence. After the embeddings are retrieved in the output layer, three token classification heads are dynamically applied. Figure 1 shows an example of one such input sequence for an input sentence. First, a subject head is applied, which marks all subjects in the input sentence. For each marked subject, the same input sentence with the marked subject is fed into the model and then into another token classification head, such as an object head. The object head marks objects, and here the input sentence with the marked subjects and objects is fed into the model again to tag predicates.

[0029] According to one embodiment of the present invention, the order of token classification heads is dynamically selected using various metrics such as entropy (for example, head predictions are ordered from minimum to maximum by entropy, which is a measure of how confident each token classification head is in its prediction). The entropy of the token classification heads, i.e., Shannon entropy, is computed by computer by aggregating only the overtokens that yield positive labels (subject, object, or predicate). The entropy is computed by computer, for example, using the softmax output from output heads, each having three classes, and then using the Shannon entropy equation. After extracting triples using a prediction order, the sentence is again input into a model that extracts triples using different prediction orders. This is preferably done multiple times, and the orders are preferably selected randomly. Triples predicted by multiple prediction orders are marked as high-confidence triples.

[0030] In the example in Figure 1, the input sentence from the completed data is passed to the transformer model. The completed data includes, for example, high-resource language data structures (e.g., (subject, predicate, object) triples) and the underlying sentences corresponding to such data structures, as well as type-switched / code-switched data structures and unstructured language text. The model maps each word in the input sentence to an embedding vector and then applies self-attention. A fully connected graph is used for the application of self-attention. The fully connected graph is generated by multiplying every token vector by every other token vector. Each token vector is a vector representing a word, and token vectors are computed by first "tokenizing" the text (dividing the sentence into words or tokens), and then mapping each word to a vector using matrix multiplication.

[0031] The output of the self-attention layer of the transformer model is a vector representation ν (i.e., an encoded vector) for each word. The sequence of vector representations is then passed to each of the three token classification heads for predicting the subject, predicate, and object. Based on the predictions, the system dynamically selects one of the token classification heads and proceeds with this head's predictions. For each head, the entropy is computed (e.g., as described above), and the order of predictions is determined by selecting the head with the lowest entropy. The predictions of the token classification heads are output in the form of actual words, and the entropy is softmaxed. Specifically, softmax is applied to the output heads, then the entropy is computed, then the order is selected, and triples are predicted based on that order. A new sentence, with all the tokens predicted by this token classification head marked, is then re-inputted into the model. The procedure is then repeated for the remaining two token classification heads. Feedback is given only during the training phase and only for the classification heads that tag subjects / objects / predicates. The system updates itself even if one of the heads tags a word, without waiting for the entire triple to be tagged.

[0032] Figure 7 shows a flowchart 700 of the operation of a multi-head self-attention transformer model, such as the one described in Figure 1 above. First, at 710, the complementary data, such as a sentence or text in a high-resource language, is input to the input layer or input encoder of the multi-head self-attention transformer model. Next, at 720, the input encoder maps each word of the input complementary data to a corresponding individual embedding vector. Next, at 730, the transformer model applies self-attention to each embedding vector to provide an encoded vector. The transformer model utilizes a fully coupled knowledge graph when performing this step. Next, at 740, the transformer model outputs the encoded vectors as a vector representation "ν" for each embedded word, thereby providing a set of encoded vectors for the input words of the complementary data.

[0033] Next, in step 750, the transformer model inputs the encoded word vectors into the token classification head layer, which is a dense (linear) layer of the transformer model. In the next step 760a, a softmax operation is performed on the output classification heads. In step 760b, an entropy metric is applied to dynamically select the initial head token tagging order, which determines whether subject token heads, predicate token heads, or object token heads will be used to tag the encoded word vectors of the initially input complementary data and to generate subject, predicate, or object triples from the initially input complementary data. In this step, head predictions are ordered from minimum to maximum by entropy (a measure of how confident the heads are in their predictions). In the next step 770, subject, predicate, or object triples are generated from this initial token classification head tagging sequence. Then, in step 780, subject, predicate, or object triples are generated from the tagged token heads and marked as high-confidence triples. Finally, in step 790, the transformer model re-inputs the completed data, including token-head tagged subject, predicate, or object triples, and repeats the operations described above to generate the remaining subject, predicate, or object triples, which may be performed multiple times in different randomly selected orders, and may be performed multiple times in this form for each (subject, predicate, object) triple type until all triples and triple types are tagged and generated. Once all triples have been generated, they are output. A data completion method for obtaining completed data according to one embodiment of the present invention comprises two parts: 1) a part for selecting English OpenIE triples containing ambiguity-free terms (see Figure 2), and 2) a part for generating OpenIE triples containing terms in a low-resource language, such as Japanese, using the English OpenIE triples (see Figure 3).

[0034] Figure 2 schematically illustrates a method and system for selecting English OpenIE triples containing deambiguated entries, starting from a publicly available OpenIE corpus. This OpenIE corpus is assumed to contain OpenIE triples with deambiguated IDs for the entries. The deambiguated ID is, in fact, a unique Wikipedia URL. Thus, such deambiguated links can be directly mapped to any Wikipedia-generated knowledge graph (e.g., DBpedia, YAGO, or Wikidata). Furthermore, such links can be directly mapped to entity mentions (e.g., the link Joe_Biden can be rewritten as the string "Joe Biden"). Embodiments of the present invention then utilize several subsets of the OpenIE corpus, namely 1) golden triples, 2) high-confidence triples, and 3) token-overlap triples. Golden triples are human-marked triples. For this, a random sample of OpenIE triples is first read. Next, a human commentator labels those triples as either “correctly extracted” or “incorrectly extracted.” Embodiments of the present invention then select only the correctly extracted triples. While using golden triples may be advantageous, the human commentary step is not required to enable the triples to be extracted according to embodiments of the present invention. High-confidence triples are triples with a confidence score > 0.95. The confidence score is obtained from the OpenIE system that generates the OpenIE corpus. Token-overlapping triples contain all tokens from the original sentence (i.e., assuming both the triple and the sentence are bag of words, tokens from the underlying sentence overlap with tokens from the triple).

[0035] According to one embodiment, all triples from a publicly available corpus are collected and a "generic filter" is applied to them. The generic filter removes triples that either 1) have no entity types in DBpedia for both terms, or 2) have a generic entity type "Thing" in DBpedia for at least one of the terms, and no terms for the other. Duplicates are filtered out.

[0036] In the example in Figure 4, the completed data is generated from English OpenIE triples. This process converts each OpenIE triple and its underlying sentence (i.e., the unstructured text / natural language sentence from which each OpenIE triple was extracted) into a triple / sentence pair. Once the triple / sentence pairs are provided, type substitution can be performed, thereby replacing one or more entities in each triple / sentence pair with another entity of the same type. Code switching then takes place, where entities from high-resource languages ​​are replaced with entities from low-resource languages. This process begins with each English OpenIE triple. Next, for the subject of each English OpenIE triple, n (e.g., 3 or 4) DBpedia entities are randomly sampled that 1) have the same entity type and 2) are available in the low-resource language DBpedia. Next, the text in the subject and the underlying sentence is replaced with the new entity of the subject (i.e., one of n randomly sampled DBpedia entities that have the same entity type as the subject and are available in the low-resource language), as well as the low-resource language text of the old entity of the object (see Figure 3). The same procedure is repeated for the object. Finally, pairs of OpenIE triples in the low-resource language and their underlying sentences are stored. This results in pairs of triples containing the terms in the low-resource language and the code-switched sentences, which the model can then extract the triples in the low-resource language.

[0037] The flowchart 800 in Figure 8 illustrates a method for extracting machine-readable data structures from unstructured, low-resource language input text. In the first step 810, a high-resource language data structure corpus containing subject, predicate or verb, and object triples is obtained. Next, in step 820, the high-resource language data structure corpus is filtered to remove certain data structures, such as mis-extracted data structures (e.g., head token-tagged) containing triples, in order to obtain a filtered high-resource language data structure corpus. Next, in step 830, entities containing a specific entity type are obtained for each entity in each data structure of the filtered corpus. Then, in step 840, a type substitution procedure is performed, thereby replacing entities in the data structures of the filtered corpus with entities of the same type in order to obtain a type-substituted data structure. Next, in step 850, a data structure code-switching process is performed, which includes replacing each entity in the type-substituted data structure with an equivalent low-resource language entity, such as a word or name, in order to obtain a code-switched sentence. Next, in step 860, the type-substituted data structures and code-switched sentences are added to the filtered corpus to provide a corpus of completed data structures after acquisition. After the completed data structure corpus is acquired, in step 870, a multi-head self-attention transformer model is trained using dynamic iterative tagging with a gradient descent algorithm. Finally, in step 880, unstructured input text of sentence forms is fed into the trained model, and machine-readable data structures are extracted from the unstructured input text.

[0038] In one embodiment, the present invention can be applied, for example, to generate a multilingual knowledge graph of corporate data. In this particular embodiment, the method can be used to generate multilingual training data in English, as well as low-resource language texts relating to corporate data such as corporate investment documents and corporate information. The result is a knowledge graph linking companies, products, investment assets, CEOs, assets, etc. The system will extract triples in different languages. Different languages ​​allow the system to take in information from various sources. This allows, for example, a bilingual investment analyst to check how companies, products, and assets are linked to each other in order to plan investment decisions.

[0039] In one embodiment, the present invention can be applied to a multilingual OpenIE system for public safety to provide, for example, an immigration control system for border management at international airports, ports, and railway crossings. Upon receiving the name and social media handle of an individual who wishes to enter or leave the border, the system invokes a multilingual OpenIE system which will extract triples from the social media feed in different languages. The triples are extracted to create a knowledge graph and will be automatically checked for terms indicating whether the person has ever endorsed, condoned, or encouraged acts of violence and illegal activity. If so, the system can automatically prohibit the person from entering or leaving the border and notify border guards to take further action. The technical advantage of the system's multilingual capabilities is that it enables the system to perform thorough background checks of individuals' social media feeds in different languages, which was not previously possible.

[0040] In one embodiment, the present invention can be applied to identify hate speech in internet articles. The spread of hate speech is a major problem on the internet. Public institutions spend considerable time and computational resources identifying and removing hate speech. This type of work is often done with the help of human workers, which is time-consuming and can have a psychological impact on the workers. In contrast, embodiments of the present invention enable the automatic extraction of triples from documents, articles, and even social media to create a knowledge graph, and such extraction is possible in multiple languages. These triples in the knowledge graph can either be automatically classified as promoting hate speech, or they can be presented to a hate speech filtering agent that eliminates the need to read the entire document and can quickly determine whether the document is hate speech by looking only at the triples and their connections. Using this system would result in significant time and computational resource savings, as well as associated costs.

[0041] In one embodiment, the present invention can be applied to summarizing text such as news articles. Figure 6 shows an exemplary application that uses an open knowledge graph to summarize a Japanese news article and compare the extracted triples. The task in this example is to extract a summary of a news article on geopolitics, and then compare the extracted triples with triples extracted from other news articles to check which facts are common and which are not. Such a system would be available for information analysts to track open-source information data obtained from publicly available news articles. The system presents analysts with structured facts from news articles and how similar they are to facts from other news articles, resulting in improved accuracy of work and significant time savings for analysts by eliminating the need to manually read or process entire articles and by allowing them to skip articles with similar facts.

[0042] In one embodiment, the present invention can be applied to extract a knowledge graph from financial news and corporate reports, for example, for the purpose of extracting corporate information, corporate assets, sales, costs, and shareholder value. The system will create a knowledge graph of how companies, products, assets, and services are linked to each other. This will be used in a link prediction algorithm to predict how a company's future performance will affect its stock valuation.

[0043] Embodiments of the present invention offer the following improvements. 1. Use a multi-head dynamic iteration procedure to extract multilingual triples. 2. Data completion using code-switched data between high-resource and low-resource languages ​​to facilitate knowledge transfer from high-resource languages ​​to low-resource languages. 3. Data completion using type substitution, which allows the model to extract entities in different languages.

[0044] In an exemplary embodiment where the high-resource language is English and the low-resource language is Japanese, the present invention provides a method for extracting triples from input text, comprising the following steps: 1. Filter Open IE triples from any publicly available OIE system. 2. From the filtered triples, retrieve the entity type in DBpedia for each triple. 3. To obtain type-substituted triples, replace the entities (subject and object) with entities of the same type. Add these triples back to the original filtered triples. 4. To obtain the code-switched sentences, replace the entities (subject and object) with their Japanese names available in DBpedia. Add these triples to the original filtered triples. 5. Using the supplemented data, train a multi-head self-attention model using dynamic iterative tagging and a gradient descent algorithm. 6. Divide an English or Japanese document into sentences, input those sentences into a trained model, and obtain triples.

[0045] Therefore, embodiments of the present invention enable the extraction of triples from text written in low-resource languages ​​such as Japanese. No existing system is capable of extracting subject, verb, and object triples in low-resource languages ​​such as Japanese. Moreover, embodiments of the present invention outperform current supervised and unsupervised English triple extraction systems through the addition of complementary data and an iterative head prediction method. Another improvement provided by embodiments of the present invention is the ability to handle code-switched sentences (i.e., sentences with mixed languages), enabling the extraction of triples even when the subject, object, and predicate are in different languages. Furthermore, embodiments of the present invention can accept sentences from more than 100 languages ​​with improved performance. For example, performance and accuracy are improved by training the model on code-switched sentences. Similarly, performance and accuracy are also improved by dynamically selecting the prediction head.

[0046] In the context describing the present invention (particularly in the context of the appended claims), the terms “a,” “an,” “the,” and “at least one,” and similar references should be interpreted as covering both singular and plural forms unless otherwise indicated herein or unless clearly contradicted by the context. The use of the term “at least one,” followed by a list of one or more items (e.g., “at least one of A and B”), should be interpreted as meaning one item (A or B) selected from the listed items, or any combination of two or more of the listed items (A and B), unless otherwise indicated herein or unless clearly contradicted by the context. The terms “comprising,” “having,” “including,” and “containing” should be interpreted as open-ended terms (i.e., “including, but not limited to”) unless otherwise noted. All methods described herein can be performed in any suitable order, unless otherwise indicated herein or unless clearly contradicted by the context. Any and all examples or illustrative language provided herein (e.g., "such as") are intended solely to better illustrate the invention and do not imply any limitation to the scope of the invention unless otherwise stated in the claims. Nothing in this specification should be construed as indicating that any non-claimed element is essential to the practice of the invention.

[0047] While embodiments of the present invention have been illustrated and described in detail in the drawings and the foregoing description, such illustrations and descriptions should be considered illustrative or illustrative, and not limiting. It will be understood that modifications and alterations are possible for those skilled in the art. Specifically, the present invention covers further embodiments, including any combination of features from different embodiments. Furthermore, the description herein characterizing the present invention illustrates one embodiment and does not necessarily illustrate all embodiments. [Explanation of symbols]

[0048] 700 flowcharts 800 Flowcharts

Claims

1. A method for extracting machine-readable data structures from unstructured first language input text, which is performed by a processor, Steps include obtaining a corpus of second language data structures, To obtain a filtered corpus of a second language data structure, the steps include filtering the corpus of the second language data structure, For each entity in each filtered second language data structure, the steps include obtaining the entity type, To generate a type-substituted data structure, the process involves performing type substitution for each retrieved entity by replacing each entity with an entity of the same type. To generate a code-switched sentence, the steps include replacing each entity contained in the filtered second language data structure or the type-substituted data structure with the corresponding first language entity, The steps include generating a completed data structure corpus, which includes combining the type-substituted data structure and the code-switched sentence with the filtered second language data structure corpus; The steps include training a multi-head self-attention transformer model using the aforementioned complementary data structure corpus with dynamic iterative tagging and gradient descent algorithms, The steps include: providing the unstructured first language input text to the trained multi-head self-attention transformer model in order to extract the machine-readable data structure; Methods that include...

2. The second language data structure includes subject, verb, and object triples, The entity type for each entity in the filtered second language data structure is obtained from at least one Open Information Exchange (OIE) system. The aforementioned supplemented data structure corpus includes a second language data structure and a first language data structure. The method according to claim 1.

3. The step of training the aforementioned multi-head self-attention transformer model is: The steps include updating the weights of the multi-head self-attention transformer model using a gradient descent algorithm and cross-entropy loss, The process involves repeating the updating step multiple times until the score obtained for the validation dataset no longer improves. The method according to claim 1, including the method described in claim 1.

4. The step of providing the unstructured first language input text to the trained multi-head self-attention transformer model is The steps include separating the unstructured first language input text into individual first language sentences, The steps include inputting the individual first-language sentences into the trained multi-head self-attention transformer model in order to obtain subject-verb-object triples, and The method according to claim 1, further comprising:

5. The method according to claim 2, wherein the entity type for the second language data structure includes subject and object classifiers.

6. The method according to claim 2, wherein the at least one OIE system includes open data, linked data, or a database of linked open data.

7. The method according to claim 1, wherein the training of the multi-head self-attention model generates a monolingual and / or multilingual knowledge graph linking entities.

8. The second language data structure includes subject, verb, and object triples, The method according to claim 3, wherein the step of updating the weights of the multi-head self-attention transformer model includes performing dynamic iterative tagging, which includes tagging the first token head of the triple subject token head, verb token head, or object token head, then tagging another token head among the subject token head, verb token head, and object token head, and then tagging the remaining token heads among the subject token head, verb token head, and object token head.

9. The second language data structure includes subject, verb, and object triples, The order of the triple dynamic head token tagging is performed dynamically using entropy, and the entropy is used to measure the prediction accuracy confidence of the head tokens from minimum to maximum using softmax calculation. The determination of the order of head token tagging is performed by aggregating only the overhead tokens that result in the triple positive head token tag. The method according to claim 3.

10. The method according to claim 9, wherein after performing the dynamic head token tagging using a first dynamic prediction sequence, at least one additional dynamic head token tagging of the triple is performed using another dynamic prediction sequence, and the triples predicted by the multiple dynamic prediction sequences are marked as high-confidence triples.

11. The method according to claim 9, wherein the dynamic head token tagging order is selected randomly.

12. The method according to claim 1, wherein the unstructured first language input text is a code-mixed sentence, and the code-mixed sentence includes a sentence in which languages ​​are mixed.

13. The method according to claim 1, wherein the unstructured first language input text comprises text of a first first language, and the extracted machine-readable data structure comprises a data structure for the first first language.

14. A non-temporary computer-readable recording medium storing processor-executable instructions that, when executed by a processor, cause the processor to perform a method for extracting machine-readable data structures from unstructured first language input text, wherein the method is Steps include obtaining a corpus of second language data structures, To obtain a filtered corpus of a second language data structure, the steps include filtering the corpus of the second language data structure, For each entity in each filtered second language data structure, the steps include obtaining the entity type, To generate a type-substituted data structure, the process involves performing type substitution for each retrieved entity by replacing each entity with an entity of the same type. To generate a code-switched sentence, the steps include replacing each entity contained in the filtered second language data structure or the type-substituted data structure with the corresponding first language entity, The steps include generating a completed data structure corpus, which includes combining the type-substituted data structure and the code-switched sentence with the filtered second language data structure corpus; The steps include training a multi-head self-attention transformer model using the aforementioned complementary data structure corpus with dynamic iterative tagging and gradient descent algorithms, The steps include: providing the unstructured first language input text to the trained multi-head self-attention transformer model in order to extract the machine-readable data structure; Non-temporary computer-readable recording media, including [specific type of media].

15. A system for extracting machine-readable data structures from unstructured first-language input text, Obtaining a corpus of second language data structures, In order to obtain a filtered corpus of a second language data structure, the corpus of the second language data structure is filtered, For each entity in each filtered second language data structure, obtain the entity type, and To generate a type-substituted data structure, type substitution is performed for each retrieved entity by replacing each entity with an entity of the same type. To generate code-switched sentences, each entity contained in the filtered second language data structure or the type-substituted data structure is replaced with the corresponding first language entity. The process includes generating a completed data structure corpus by combining the type-substituted data structures and the code-switched sentences with the filtered second language data structure corpus, Using the aforementioned complementary data structure corpus, a multi-head self-attention transformer model is trained using dynamic iterative tagging and gradient descent algorithms. In order to extract the machine-readable data structure, the unstructured first language input text is fed to the trained multi-head self-attention transformer model. A system comprising one or more processors configured to perform the following tasks.

Images