A training method and device of an open information extraction model

Abstract

The application provides a training method and device of an open information extraction model, comprising: obtaining a target data set with natural language sentences as samples; generating paraphrases of each natural language sentence in the target data set; performing structured knowledge recovery on the paraphrases of each natural language sentence in the target data set to obtain structured knowledge corresponding to each natural language sentence in the target data set; constructing a first data set with the paraphrases and the structured knowledge corresponding to all natural language sentences in the target data set; and training the open information extraction model by using the first data set and the target data set in a noise reduction training mode. The application constructs a syntax-robust training framework based on paraphrase generation and structured knowledge recovery, so that the open information extraction model can be trained on a data set with sufficient and accurate syntax distribution to adapt to real-world scenarios.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to a training method and device of an open information extraction model. BACKGROUND

[0002] Open information extraction is a technology of converting a given natural language text into a structured knowledge N-tuple composed of <argument1, predicate, argument2, …, argumentn>. The open information extraction model has outstanding independence and scalability in structured knowledge N-tuple extraction, and has become a cornerstone of many natural language processing applications such as automatic text summarization and question answering systems.

[0003] However, early open information extraction models use rule-based or statistical methods to extract structured knowledge N-tuples, which strongly depend on the syntactic features existing in the data set and are difficult to generalize to unknown syntactic structures in the real world. Open information extraction models constructed by drawing lessons from deep learning technology (generally divided into two categories: sequence labeling-based open information extraction models and sequence generation-based open information extraction models) are all limited by the partial observability of syntactic distribution in the training set, resulting in a significant decline in model performance in real-world scenarios with diverse syntactic distribution.

[0004] Therefore, there is an urgent need to train an open information extraction model that is always robust on a data set with sufficient syntactic distribution. SUMMARY

[0005] To solve or partially solve the above technical problems, the present application provides a training method and device of an open information extraction model, which constructs a syntactically robust training framework based on paraphrase generation and structured knowledge recovery, so that the open information extraction model can be trained on a data set with sufficient and accurate syntactic distribution to adapt to real-world scenarios.

[0006] In a first aspect, the present application provides a training method of an open information extraction model, which comprises:

[0007] obtaining a target data set with natural language sentences as samples;

[0008] generating a paraphrase sentence for each natural language sentence in the target data set;

[0009] performing structured knowledge recovery on the paraphrase sentence of each natural language sentence in the target data set to obtain the corresponding structured knowledge of each natural language sentence in the target data set;

[0010] constructing a first data set with the paraphrase sentences and the structured knowledge corresponding to all natural language sentences in the target data set;

[0011] The open information extraction model is trained in a denoising manner by using the first data set and the target data set.

[0012] According to the training method of the open information extraction model provided by the application, the paraphrase sentence of each natural language sentence in the target data set is generated, and the method comprises the following steps:

[0013] For each natural language sentence in the target data set, a sequence formed by the natural language sentence, a syntactic tree corresponding to the natural language sentence and a target syntactic tree is input into a syntactically controllable paraphrase generation model to obtain a sentence output by the paraphrase generation model and complying with the target syntactic tree.

[0014] The sentence is taken as the paraphrase sentence of the natural language sentence.

[0015] The paraphrase generation model is trained by using a parallel corpus ParaNMT-small.

[0016] Each sample in the parallel corpus ParaNMT-small is composed of a natural language sentence and a corresponding paraphrase sentence.

[0017] According to the training method of the open information extraction model provided by the application, the syntactic tree corresponding to the natural language sentence is determined in the following manner:

[0018] The natural language sentence is subjected to syntactic analysis and bracket tree form representation to obtain the syntactic tree corresponding to the natural language sentence.

[0019] The target syntactic tree is determined in the following manner:

[0020] The syntactic tree corresponding to the natural language sentence and the syntactic tree corresponding to the paraphrase sentence in each sample of the parallel corpus ParaNMT-small are determined.

[0021] After the syntactic tree corresponding to the natural language sentence and the syntactic tree corresponding to the paraphrase sentence in each sample of the parallel corpus ParaNMT-small are subjected to truncation processing of only retaining the first three layers, the syntactic tree pair corresponding to each sample of the parallel corpus ParaNMT-small is generated by combination.

[0022] The syntactic tree pair set is composed of the syntactic tree pairs corresponding to all samples of the parallel corpus ParaNMT-small, the first syntactic tree set is composed of the syntactic trees corresponding to the natural language sentences in all samples of the parallel corpus ParaNMT-small, and the second syntactic tree set is composed of the syntactic trees corresponding to the paraphrase sentences in all samples of the parallel corpus ParaNMT-small.

[0023] selecting M syntax trees with the highest similarity to the syntax tree corresponding to the natural language sentence from the first syntax tree set, and taking the M syntax trees as source syntax trees;

[0024] determining a distribution function corresponding to each source syntax tree according to the frequency of occurrence of a syntax tree pair formed by each source syntax tree and each syntax tree in the second syntax tree set in the syntax tree pair set;

[0025] sampling K syntax trees from the distribution function corresponding to each source syntax tree to obtain M×K syntax trees;

[0026] taking the M×K syntax trees as the target syntax trees;

[0027] wherein the distribution function corresponding to each source syntax tree is as follows:

[0028]

[0029] In the above formula, is the source syntax tree, is a syntax tree in the second syntax tree set, is the jth syntax tree in the second syntax tree set, is the frequency of occurrence in the syntax tree pair set, is the frequency of occurrence in the syntax tree pair set.

[0030] According to the training method of the open information extraction model provided by the application, the paraphrase of each natural language sentence in the target data set is structured and knowledge is recovered to obtain the structured knowledge corresponding to each natural language sentence in the target data set, which comprises the following steps:

[0031] The paraphrase of each natural language sentence in the target data set is recovered by using a semantic matching method, and a first sentence is obtained;

[0032] The first sentence is recovered by using a syntax tree walking method, and a second sentence is obtained;

[0033] The second sentence is recovered by using a predicate prediction method, and the structured knowledge corresponding to the natural language sentence is obtained.

[0034] According to the training method of the open information extraction model provided by the application, the paraphrase of each natural language sentence in the target data set is recovered by using a semantic matching method, and a first sentence is obtained, which comprises the following steps:

[0035] obtaining a first vectorized embedding representation corresponding to the natural language sentence and a second vectorized embedding representation corresponding to the paraphrased sentence;

[0036] For each of the triples in the natural language sentence, a cosine similarity between an argument / predicate in the triple and each word in the paraphrased sentence is calculated according to the first vectorized embedding representation and the second vectorized embedding representation to obtain a similarity score sequence corresponding to the argument / predicate in the triple;

[0037] A position interval in which similarity scores in the similarity score sequence corresponding to the argument / predicate in the triple are successively greater than a preset threshold is taken as a candidate position of the argument / predicate in the triple;

[0038] According to the candidate position of the argument in the triple and the candidate position of the predicate in the triple, a position of the triple is located based on the highest overall similarity and no position overlap;

[0039] According to the located position of each of the triples in the natural language sentence, the first sentence is generated.

[0040] According to the training method of the open information extraction model provided by the present application, the argument in the triple corresponding similarity score sequence is calculated according to the following formula:

[0041]

[0042] the predicate in the triple corresponding similarity score sequence c r is calculated according to the following formula:

[0043] wherein, is a part corresponding to a jth word of the natural language sentence in the first vectorized embedding representation, h t is the second vectorized embedding representation, is a first word of the argument in the natural language sentence, is a last word of the argument in the natural language sentence, is a first word of the predicate in the natural language sentence, is a last word of the predicate in the natural language sentence.

[0044] The training method of the open information extraction model provided by the application adopts a syntax tree walk method to recover arguments of the first sentence and obtain a second sentence, and comprises the following steps:

[0045] For each word of each argument in the first sentence, a postorder traversal is performed on the target syntax tree to find a subtree with NP, QP or NX as the root and containing the word, and the subtree is recorded as a subtree corresponding to the word;

[0046] The subtrees corresponding to all the words in the argument are grouped into a third set;

[0047] A word sequence corresponding to each subtree in the third set is taken as a replacement option of the argument, and a replacement set of the argument is generated;

[0048] Based on the replacement sets of all the arguments in the first sentence, the arguments in the first sentence are optimized based on the highest overall similarity and no position overlap to obtain the second sentence.

[0049] The training method of the open information extraction model provided by the application adopts a predicate prediction method to recover predicates of the second sentence and obtain structured knowledge corresponding to the natural language sentence, and comprises the following steps:

[0050] The second sentence and all the arguments in the second sentence are input into a pre-constructed T5 model to obtain a predicate output by the T5 model; and the predicate is used to optimize the second sentence to obtain structured knowledge corresponding to the natural language sentence;

[0051] The T5 model is trained by using a predicate prediction data set;

[0052] The predicate prediction data set is constructed based on the target data set and takes a natural language sentence, all the arguments in the natural language sentence, and a predicate in the natural language sentence as sample data.

[0053] The training method of the open information extraction model provided by the application adopts a noise reduction training method to train the open information extraction model by using the first data set and the target data set, and comprises the following steps:

[0054] Word importance score labels and argument random masks are added to the paraphrased sentences in the first data set to obtain a second data set;

[0055] Argument random masks are added to the target data set;

[0056] The union of the target data set with the argument random masks and the second data set is taken as an expansion set;

[0057] An open information extraction model is trained by using the extended set and according to a training target.

[0058] The training target expression is as follows:

[0059]

[0060] In the above formula, r1, …, r m are triples to be extracted, s is a sentence in the extended set, p d is a sample distribution on the extended set, p θ is a neural network model.

[0061] In a second aspect, the present application provides a device for training an open information extraction model, which comprises:

[0062] An acquisition module is configured to acquire a target data set with natural language sentences as samples.

[0063] A paraphrase generation module is configured to generate a paraphrase of each natural language sentence in the target data set.

[0064] A structured knowledge recovery module is configured to perform structured knowledge recovery on the paraphrase of each natural language sentence in the target data set to obtain structured knowledge corresponding to each natural language sentence in the target data set.

[0065] A first data set generation module is configured to construct a first data set with the paraphrases and the structured knowledge of all natural language sentences in the target data set.

[0066] A noise reduction training module is configured to train an open information extraction model by using the first data set and the target data set and adopting a noise reduction training method.

[0067] The present application provides a training method and device for an open information extraction model, which acquires a target data set with natural language sentences as samples, generates a paraphrase of each natural language sentence in the target data set to form a paraphrase set with sufficient syntax distribution, performs structured knowledge recovery on the paraphrase of each natural language sentence in the target data set to obtain structured knowledge corresponding to each natural language sentence in the target data set, constructs a first data set with the paraphrases and the structured knowledge of all natural language sentences in the target data set, and trains an open information extraction model by using the first data set and the target data set and adopting a noise reduction training method. The present application constructs a syntax-robust training framework based on paraphrase generation and structured knowledge recovery, solves the problem of observable syntax in the training set of an existing model, avoids the loss of structured knowledge caused by the generation of a paraphrase set, and enables the open information extraction model to be trained on a data set with sufficient and accurate syntax distribution to adapt to real-world scenarios. Attached Figure Description

[0068] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0069] Figure 1 This is a flowchart illustrating a training method for an open information extraction model provided by the present invention.

[0070] Figure 2 This is a training framework diagram for the syntactically robust open information extraction model provided by the present invention;

[0071] Figure 3 This is a schematic diagram of the structure of a training device for an open information extraction model provided by the present invention;

[0072] Figure 4 This is a schematic diagram of the structure of an electronic device that implements the training method for an open information extraction model provided by the present invention;

[0073] Figure label:

[0074] 410: Processor; 420: Communication interface; 430: Memory; 440: Communication bus. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0076] The following definitions are provided for abbreviations and key terms in this field:

[0077] Open information extraction: For a given natural language text, automatically extract structured knowledge N-tuples consisting of <argument 1, predicate, argument 2, ..., argument n>.

[0078] Predicate: An element that constitutes a fact in a structured knowledge N-tuple, representing the action or behavior of that fact. In particular, a predicate may be a word or phrase in the original sentence, or it may be a word or phrase expressing an action or subordination that does not appear in the original sentence.

[0079] Argument: an element that constitutes a structured knowledge N-tuple fact, representing the initiating object, the taking object and the associated object of the fact. In particular, argument 1 represents the initiating object of the fact, and arguments 2 to n represent the taking object and the associated object of the fact in turn.

[0080] Robustness: the consistency of the effect of a model in different environmental scenarios. In particular, if a model shows consistent effects in different environmental scenarios, the present invention refers to the model as having strong robustness.

[0081] Paraphrase generation: transforming the expression form of a given natural language sentence without changing its semantic, including syntax transformation and word transformation, etc.

[0082] Span: a continuous interval for a given natural language sentence, corresponding to a sequence of words.

[0083] False positive: for a binary classification sample with a negative true class label, the model prediction result is positive, which is called false positive.

[0084] Syntactic distribution: refers to the position where a word or phrase can or cannot appear, such as nouns often appear in the subject, object, attribute, and predicate of a judgment sentence, and sometimes in the position of an adverbial.

[0085] Partially observable syntax, that is, the syntactic distribution on the data set is a partial observation of the real world syntactic distribution, which far from covers the entire hypothesis space.

[0086] The following will be described in combination Figures 1-4 A training method and device of an open information extraction model are described.

[0087] In the first aspect, in order to build an open information extraction model that is always robust, the present invention needs to train the model on a data set with sufficient and accurate syntactic distribution. However, manually annotating data with syntactic diversity in an artificial way is extremely expensive and impractical; due to the uncertainty of the types of arguments and predicates, it is also infeasible to automatically annotate data in a remote supervision manner for open information extraction. Therefore, obtaining a diverse and accurate data set that meets the syntactic distribution assumption is a highly challenging problem. To this end, the present invention provides a training method of an open information extraction model, as described in the following Figure 1 The method comprises:

[0088] S11, obtaining a target data set with natural language sentences as samples;

[0089] S12, generating a paraphrase sentence for each natural language sentence in the target data set;

[0090] Humans learn grammar by learning different expressions of the same semantics, for example: "Jobs founded Apple in 1976" and "Apple was founded by Jobs in 1976" express the same semantics. Therefore, on a conventional scale, paraphrase generation is performed on natural language sentences in a target data set D, so as to construct a paraphrase set P with syntactic diversity (or a perfect syntactic distribution). An open information extraction model trained by using the paraphrase set P can solve the problem of syntactic observability of an existing open information extraction model.

[0091] Specifically, S12 comprises:

[0092] S12-1, for each natural language sentence in the target data set, inputting a sequence formed by the natural language sentence, a syntactic tree corresponding to the natural language sentence and a target syntactic tree into a syntactically controllable paraphrase generation model, to obtain a sentence output by the paraphrase generation model and complying with the target syntactic tree;

[0093] S12-2, taking the sentence as a paraphrase sentence of the natural language sentence;

[0094] The paraphrase generation model is trained by using a parallel corpus ParaNMT-small.

[0095] Each sample in the parallel corpus ParaNMT-small is composed of a natural language sentence and a corresponding paraphrase sentence.

[0096] That is, the syntactically controllable paraphrase generation model AESOP is adopted to complete the process, and the input of the paraphrase generation model AESOP is a sequence "source sentence <sep>source full syntactic parse <sep>target syntactic parse", outputting a target sentence (a paraphrase of the natural language sentence) adhering to a target syntactic parse <sep>paraphrase”.

[0097] Further, the syntax tree corresponding to the natural language sentence is determined in the following manner:

[0098] The syntax analysis and bracket tree form representation are performed on the natural language sentence to obtain the syntax tree corresponding to the natural language sentence.

[0099] Preferably, for the data set D, the present application first uses the Stanford CoreNLP tool to obtain the parse tree of all natural language sentences and linearizes them into bracket tree form, where is the αth natural language sentence in the data set D, α ∈ (1 ~ N), N is the number of natural language sentences contained in the data set D.

[0100] The determination step of the target syntax tree is as follows:

[0101] Step A: Determine the syntax tree corresponding to the natural language sentence and the syntax tree corresponding to the paraphrase sentence in each sample of the parallel corpus ParaNMT-small;

[0102] Step B: After performing the truncation processing of retaining only the first three layers on the syntax tree corresponding to the natural language sentence and the syntax tree corresponding to the paraphrase sentence in each sample of the parallel corpus ParaNMT-small, the syntax tree pair corresponding to each sample of the parallel corpus ParaNMT-small is generated by combination;

[0103] Step C: The syntax tree pair set is composed of the syntax tree pairs corresponding to all samples of the parallel corpus ParaNMT-small, the first syntax tree set is composed of the syntax trees corresponding to the natural language sentences in all samples of the parallel corpus ParaNMT-small, and the second syntax tree set is composed of the syntax trees corresponding to the paraphrase sentences in all samples of the parallel corpus ParaNMT-small;

[0104] In actual operation, the present application can collect the syntax tree pair set truncated to 3 layers (i.e. only retaining the root node, the first layer, the second layer and the third layer) from the parallel corpus ParaNMT-50M where is the syntax tree pair corresponding to the βth natural language sentence-paraphrase sentence in the parallel corpus ParaNMT-50M, β ∈ (1 ~ |P|), |P| is the number of natural language sentences-paraphrase sentences contained in the parallel corpus ParaNMT-50M. And the first syntax tree set is composed of and the second syntax tree set is composed of .

[0105] Step D: selecting M syntax trees with the highest similarity to the syntax tree corresponding to the natural language sentence from the first syntax tree set, and taking the M syntax trees as source syntax trees

[0106] Here, the similarity of the syntax tree corresponding to the natural language sentence can be represented by a similarity score.

[0107] Step E: determining a distribution function corresponding to each source syntax tree according to the frequency of each source syntax tree pair in the syntax tree pair set, wherein the distribution function corresponding to each source syntax tree is As follows:

[0108]

[0109] In the above formula, is the source syntax tree, is the syntax tree in the second syntax tree set, is the jth syntax tree in the second syntax tree set, is the frequency of the syntax tree pair in the syntax tree pair set, is the frequency of the syntax tree pair in the syntax tree pair set;

[0110] Step F: sampling K syntax trees from the distribution function corresponding to each source syntax tree to obtain MxK syntax trees, and taking each of the MxK syntax trees as the target syntax tree.

[0111] When sampling K syntax trees from the distribution function corresponding to any source syntax tree, the syntax trees belonging to the second syntax tree set in the syntax tree pair corresponding to the K largest values in the distribution function can be selected, or random sampling can be performed.

[0112] It can be understood that each D generates KM paraphrased sentences, in order to balance the quality and quantity of the generated paraphrased sentences, the present application sets K and M to 5 and 1 respectively. So that the number of sentences in the paraphrased sentence candidate set P is about 5 times that of D.

[0113] S13, performing structured knowledge recovery on the paraphrased sentences of each natural language sentence in the target data set to obtain structured knowledge corresponding to each natural language sentence in the target data set;

[0114] The paraphrased set generation faces an internal problem of knowledge deformation, that is, the loss of N-tuple structured knowledge in the original sample after paraphrasing, therefore, how to find the mapping of structured knowledge before and after paraphrasing is a key, and the purpose of S13 is to recover the deformed structured knowledge in the paraphrased sentence.

[0115] Specifically, the S13 comprises:

[0116] S13-1, using a semantic matching manner, performing argument recovery on the paraphrase of each natural language sentence in the target data set to obtain a first sentence;

[0117] S13-2, using a syntax tree walking manner, performing argument recovery on the first sentence to obtain a second sentence;

[0118] S13-3, using a predicate prediction manner, performing predicate recovery on the second sentence to obtain the structured knowledge corresponding to the natural language sentence.

[0119] There are two challenges in knowledge recovery: first, the arguments in the open information extraction target N-tuple are usually long word sequences, which can be rearranged and recombined into new unknown forms; second, the syntactic transformation of the paraphrase may change the tense and voice of the sentence. Therefore, the knowledge recovery is divided into two stages of argument recovery and predicate recovery as a whole.

[0120] In detail, the application first locates the position of the argument based on semantic matching; then perfects the argument based on syntax tree walking; and uses a T5 model to predict the predicate of the recovered pair of arguments.

[0121] In the argument recovery stage, the application locuses the position of the argument by semantic similarity matching, and this process is realized by using a BERT (pre-trained language model).

[0122] S13-1 comprises:

[0123] S13-1-1, obtaining a first vectorized embedding representation h s ∈R |s|×d and a second vectorized embedding representation h t ∈R |t|×d corresponding to the paraphrase;

[0124] S13-1-2, for each triple in the natural language sentence, according to the first vectorized embedding representation and the first vectorized embedding representation, calculating the cosine similarity between the argument / predicate in the triple and each word in the paraphrase to obtain a similarity score sequence corresponding to the argument / predicate in the triple;

[0125] That is, the similarity score sequence corresponding to the argument / predicate in the triple is calculated as follows:

[0126]

[0127] ​the predicate in the triple the corresponding sequence of similarity scores c r is calculated as follows:

[0128] wherein, is the corresponding part of the j-th word of the natural language sentence in the first vectorized embedding representation, h t is the corresponding part of the j-th word of the natural language sentence in the second vectorized embedding representation, is the first word of the argument in the natural language sentence, is the last word of the argument in the natural language sentence, is the first word of the predicate in the natural language sentence, is the last word of the predicate in the natural language sentence.

[0129] For example, for the triple and the sequence of similarity scores c r and

[0130] S13-1-3, the position interval in which the similarity scores in the sequence of similarity scores corresponding to the argument / predicate in the triple are successively greater than a preset threshold τ is taken as the candidate position of the argument / predicate in the triple;

[0131] For example, the candidate set of candidate positions of the argument is obtained The candidate set of candidate positions of the predicate is obtained The candidate set of candidate positions of the argument is obtained

[0132] S13-1-4, according to the candidate positions of the argument in the triple and the candidate positions of the predicate in the triple, the position of the triple is located according to the principle of the highest overall similarity and no position overlap;

[0133] It can be understood that each candidate position has its corresponding similarity value, and the best position combination of the argument and , i.e. the best position of the triple , can be selected according to the principle of the highest overall similarity and no position overlap.

[0134] S13-1-4, generating the first sentence according to the position of each triple in the natural language sentence positioned.

[0135] By applying this algorithm to the set P, a dataset D containing robust samples can be obtained P . The new dataset obtained after expansion is denoted as D Φ = D U D P .

[0136] Although the triples obtained by semantic similarity matching are accurate in position, the arguments recovered are often incomplete because words such as prepositions and adverbs are difficult to effectively obtain through similarity matching. Therefore, we propose to further improve the recovered arguments by means of syntactic tree traversal.

[0137] S13-2, comprising:

[0138] S13-2-1, for each word of each argument in the first sentence, performing a post-order traversal on the target syntactic tree to find a subtree rooted at NP, QP or NX and containing the word, and denoted as the subtree corresponding to the word;

[0139] S13-2-2, grouping all the subtrees corresponding to the words in the argument into a third set;

[0140] S13-2-3, taking the sequence of words corresponding to each subtree in the third set as a replacement option for the argument, and generating a replacement set for the argument;

[0141] S13-2-4, optimizing the arguments in the first sentence based on the replacement sets of all arguments in the first sentence, with the highest overall similarity and no position overlap as the basis, to obtain the second sentence.

[0142] It can be understood that each replacement option has its corresponding position interval and similarity value. With the highest overall similarity and no position overlap as the basis, the arguments can be optimized from the whole (first sentence).

[0143] Since paraphrase generation can change the tense and voice of the predicate, the same predicate recovery algorithm as argument recovery cannot be used. Considering that many predicates are not continuous word sequences in the sentence, the T5 model is used to recover the predicate in the paraphrase sentence.

[0144] Specifically, S13-3, comprising:

[0145] S13-3-1, inputting the second sentence and all arguments in the second sentence into a pre-constructed T5 model to obtain the predicate output by the T5 model;

[0146] For example, the second sentence and all arguments in the second sentence can be represented in the form of "source sentence, argument1, argument2";

[0147] S13-3-2, optimizing the second sentence by using the predicate, to obtain the structured knowledge corresponding to the natural language sentence;

[0148] The T5 model is trained by using a predicate prediction data set.

[0149] The predicate prediction data set is constructed on the basis of the target data set, and takes a natural language sentence, all arguments in the natural language sentence, and a predicate in the natural language sentence as sample data.

[0150] The application restores the predicate in the above argument recovery algorithm based on the trained T5 model, and finally obtains a complete restored data set D P′ . The extended set is referred to as D Ψ =D∪D p ′.

[0151] S14, constructing a first data set by using the paraphrase sentences and the structured knowledge corresponding to all natural language sentences in the target data set;

[0152] S15, training an open information extraction model by using the first data set and the target data set in a noise reduction training manner.

[0153] Specifically, the S15 comprises:

[0154] S14-1, marking the paraphrase sentences in the first data set with word importance scores and argument random masks, to obtain a second data set;

[0155] For a triple (a1, p r ,a2), the word located by the semantic similarity matching is set to an importance score based on the inverse of the semantic similarity matching mode, and for those words recovered by the syntax tree walk, the inverse of the average semantic similarity of other words in the triple (a1, p r ,a2) is set.

[0156] S14-2, performing argument random masking on the target data set;

[0157] Since the recovered long argument span can bring false positive noise of some words, a simple and effective masking strategy is adopted during training to ignore the negative impact of the wrong words. For example, 15% are randomly selected for masking to ignore the gradient calculation of these words during training. Since the predicate is generally short and tends to have no noise, we only apply the masking strategy to the argument to reduce noise.

[0158] S14-3, the union set of the argument randomly masked target data set and the second data set as an expansion set;

[0159] S14-4, using the expansion set, and training an open information extraction model according to the training target;

[0160] Wherein, the training target, according to the sample distribution on the expansion set, maximizes the expectation of the following log-likelihood function:

[0161]

[0162] In the above formula, r1,…,r m are the triplets to be extracted, s is the sentence in the expansion set, p d is the sample distribution on the expansion set, which can be understood as the joint distribution probability of m triplets; p θ is a neural network model, which either adopts a sequence labeling paradigm to predict a classification label sequence on the input sequence, or adopts a sequence generation paradigm to generate target triplets word by word.

[0163] In summary, the actual operation of the syntax robust open information extraction model training framework is as shown in the figure. Figure 2

[0164] The present application provides a kind of training method of open information extraction model, based on using paraphrase generation mode solves the syntax part observable problem of existing model training set;To deal with the internal problem of paraphrase generation, a set of argument recovery algorithm based on semantic similarity matching and syntax tree walk, and predicate recovery method based on T5 model are proposed, which avoids the structured knowledge loss phenomenon caused by paraphrase set generation;The comprehensive use of paraphrase generation and knowledge recovery makes the open information extraction model be able to train on the data set with sufficient and accurate syntax distribution, to adapt to real world scene.

[0165] In order to explore the robustness of open information extraction model, we also construct a syntax diverse validation set consistent with real world assumptions. In addition to the standard open information extraction validation set CaRB, we also construct a syntax diverse validation set CaRB-AutoPara based on the above paraphrase generation and knowledge recovery algorithm to verify the robustness of the open information extraction model. ​

[0166] The experimental results show that, compared with the baseline model (trained with dataset D, or trained with dataset D Φ The syntactic robust training framework proposed by the present application can generally improve the effect of the model on the syntactic in-distribution dataset CaRB compared with the baseline model. At the same time, since the validation set CaRB and the training set OpenIE4 are both derived from the Wikipedia data source, their syntactic distributions are similar. When we conduct experiments on the syntactically diverse validation set CaRB-AutoPara constructed in accordance with the real-world assumption, we find that the effect of the baseline model has dropped significantly, while the model trained by the syntactic robust training framework proposed by the present application exhibits consistent and leading results.

[0167] In a second aspect, the training device of the open information extraction model is described, and the training device of the open information extraction model described below can be correspondingly referred to the training method of the open information extraction model described above. Figure 3 The structural diagram of the open information extraction model is exemplified as shown in Figure 3 The device comprises:

[0168] The acquisition module 21 is configured to acquire a target dataset with natural language sentences as samples.

[0169] The paraphrase generation module 22 is configured to generate a paraphrase of each natural language sentence in the target dataset.

[0170] The structured knowledge recovery module 23 is configured to perform structured knowledge recovery on the paraphrase of each natural language sentence in the target dataset to obtain the structured knowledge corresponding to each natural language sentence in the target dataset.

[0171] The first dataset generation module 24 is configured to construct a first dataset with the paraphrases and the structured knowledge of all natural language sentences in the target dataset.

[0172] The noise reduction training module 25 is configured to train the open information extraction model in a noise reduction training manner by using the first dataset and the target dataset.

[0173] The present application provides a training device of an open information extraction model, which solves the syntactic partial observability problem of the existing model training set by using a paraphrase generation method. In order to deal with the internal problem of paraphrase generation, a set of argument recovery algorithm based on semantic similarity matching and syntactic tree walking, and a predicate recovery method based on T5 model are proposed, which avoids the loss of structured knowledge caused by paraphrase set generation. The comprehensive use of paraphrase generation and knowledge recovery enables the open information extraction model to be trained on a dataset with sufficient and accurate syntactic distribution to adapt to real-world scenarios.

[0174] On the basis of each of the above embodiments, as an optional embodiment, the paraphrase generation module 22 is specifically configured to:

[0175] for each natural language sentence in the target data set, inputting a sequence formed by the natural language sentence, a syntax tree corresponding to the natural language sentence, and a target syntax tree into a syntax controllable paraphrase generation model, to obtain a sentence output by the paraphrase generation model and complying with the target syntax tree;

[0176] taking the sentence as a paraphrase sentence of the natural language sentence;

[0177] The paraphrase generation model is trained using a parallel corpus ParaNMT-small.

[0178] Each sample in the parallel corpus ParaNMT-small is composed of a natural language sentence and a corresponding paraphrase sentence.

[0179] On the basis of each of the above embodiments, as an optional embodiment, the syntax tree corresponding to the natural language sentence is determined in the following manner:

[0180] performing syntax analysis and bracket tree form representation on the natural language sentence to obtain a syntax tree corresponding to the natural language sentence;

[0181] The target syntax tree is determined in the following manner:

[0182] determining a syntax tree corresponding to a natural language sentence and a syntax tree corresponding to a paraphrase sentence in each sample of the parallel corpus ParaNMT-small;

[0183] After performing truncation processing on the syntax tree corresponding to the natural language sentence and the syntax tree corresponding to the paraphrase sentence in each sample of the parallel corpus ParaNMT-small by retaining only the first three layers, combining to generate a syntax tree pair corresponding to each sample of the parallel corpus ParaNMT-small;

[0184] a syntax tree pair set composed of syntax tree pairs corresponding to all samples of the parallel corpus ParaNMT-small, a first syntax tree set composed of syntax trees corresponding to natural language sentences in all samples of the parallel corpus ParaNMT-small, and a second syntax tree set composed of syntax trees corresponding to paraphrase sentences in all samples of the parallel corpus ParaNMT-small;

[0185] selecting M syntax trees with the highest similarity to the syntax tree corresponding to the natural language sentence from the first syntax tree set, and taking the M syntax trees as source syntax trees;

[0186] determining a distribution function corresponding to each source-side syntax tree according to a frequency of occurrence of a syntax tree pair formed by each source-side syntax tree and each syntax tree in the second syntax tree set in the syntax tree pair set;

[0187] sampling K syntax trees from the distribution function corresponding to each source-side syntax tree to obtain MxK syntax trees;

[0188] taking the MxK syntax trees as the target syntax tree;

[0189] wherein the distribution function corresponding to each source-side syntax tree is as follows:

[0190]

[0191] In the above formula, is the source-side syntax tree, is a syntax tree in the second syntax tree set, is the jth syntax tree in the second syntax tree set, is is the frequency of occurrence of the syntax tree pair in the syntax tree pair set, is is the frequency of occurrence of the syntax tree pair in the syntax tree pair set.

[0192] On the basis of each of the above embodiments, as an optional embodiment, the structured knowledge recovery module 23 specifically comprises:

[0193] a semantic matching unit, configured to perform argument recovery on a paraphrase of each natural language sentence in the target data set in a semantic matching manner to obtain a first sentence;

[0194] a syntax tree walking unit, configured to perform argument recovery on the first sentence in a syntax tree walking manner to obtain a second sentence;

[0195] a predicate prediction unit, configured to perform predicate recovery on the second sentence in a predicate prediction manner to obtain the structured knowledge corresponding to the natural language sentence.

[0196] On the basis of each of the above embodiments, as an optional embodiment, the semantic matching unit comprises:

[0197] a vectorized embedding representation subunit, configured to obtain a first vectorized embedding representation corresponding to the natural language sentence and a second vectorized embedding representation corresponding to the paraphrase;

[0198] The similarity calculation sub-unit is configured to calculate, for each of the triples in the natural language sentence, a cosine similarity between the argument / predicate in the triple and each word in the paraphrased sentence according to the first vectorized embedding representation and the first vectorized embedding representation to obtain a similarity score sequence corresponding to the argument / predicate in the triple;

[0199] The candidate position determination sub-unit of the argument / predicate in the triple is configured to take a position interval in which the similarity scores in the similarity score sequence corresponding to the argument / predicate in the triple are successively greater than a preset threshold as a candidate position of the argument / predicate in the triple.

[0200] The positioning sub-unit is configured to position the position of the triple according to the candidate position of the argument in the triple and the candidate position of the predicate in the triple, with the highest overall similarity and no position overlap as the basis.

[0201] The first sentence generation sub-unit is configured to generate the first sentence according to the position of each of the triples in the natural language sentence that is positioned.

[0202] On the basis of each of the above embodiments, as an optional embodiment, the candidate position of the argument in the triple is determined according to the similarity score sequence corresponding to the argument in the triple. The corresponding similarity score sequence c is calculated according to the following formula:

[0203]

[0204] The corresponding similarity score sequence c of the predicate in the triple is calculated according to the following formula: r

[0205] wherein, is a portion corresponding to the jth word in the natural language sentence in the first vectorized embedding representation, h t is the second vectorized embedding representation, is the first word of the argument in the natural language sentence, is the last word of the argument in the natural language sentence, is the first word of the predicate in the natural language sentence, is the last word of the predicate in the natural language sentence.

[0206] On the basis of each of the above embodiments, as an optional embodiment, the syntax tree walking sub-unit specifically comprises:

[0207] ​The sub-unit is used to perform a post-order traversal of the target syntax tree for each word of each argument in the first statement to find the subtree rooted at NP, QP or NX that contains the word, and record it as the subtree corresponding to the word.

[0208] An integration subunit is used to group all the subtrees corresponding to the words in the argument into a third set;

[0209] The argument replacement set determines the subunit, which is used to generate the argument replacement set by taking the word sequence corresponding to each subtree in the third set as a replacement option for the argument;

[0210] The argument optimization subunit is used to optimize the arguments in the first statement based on the replacement set of all arguments in the first statement, with the highest overall similarity and no positional overlap, to obtain the second statement.

[0211] Based on the above embodiments, as an optional embodiment, the predicate prediction unit is used for:

[0212] The second statement and all arguments in the second statement are input into a pre-built T5 model to obtain the predicates output by the T5 model; the second statement is optimized using the predicates to obtain the structured knowledge corresponding to the natural language statement.

[0213] The T5 model is trained using a predicate prediction dataset.

[0214] The predicate prediction dataset is constructed based on the target dataset, and uses natural language sentences, all arguments in natural language sentences, and predicates in natural language sentences as sample data.

[0215] Based on the above embodiments, as an optional embodiment, the noise reduction training module 25 includes:

[0216] The importance score labeling and first masking unit is used to label the word importance scores and randomize the arguments of the paraphrased sentences in the first dataset to obtain the second dataset.

[0217] The second masking unit is used to perform argument random masking on the target dataset;

[0218] The union of the target dataset with the argument random mask and the second dataset is used as the extended set;

[0219] The training unit is used to train the open information extraction model using the extended set and according to the training objective.

[0220] The training objective expression is as follows:

[0221]

[0222] In the above formula, r1, …, r m are triples to be extracted, s is a sentence in the expanded set, p d is a sample distribution on the expanded set, p θ is a neural network model.

[0223] In a third aspect, Figure 4 An entity structure diagram of an electronic device is exemplified, as Figure 4 shown, the electronic device can include: a processor 410, a communications interface 420, a memory 430 and a communications bus 440, wherein the processor 410, the communications interface 420, the memory 430 complete mutual communication through the communications bus 440. The processor 410 can invoke the logical instructions in the memory 430 to execute the training method of the open information extraction model, the method comprising: obtaining a target data set with natural language sentences as samples; generating a paraphrase sentence for each natural language sentence in the target data set; performing structured knowledge recovery on the paraphrase sentence of each natural language sentence in the target data set to obtain the structured knowledge corresponding to each natural language sentence in the target data set; constructing a first data set with the paraphrase sentences and structured knowledge corresponding to all natural language sentences in the target data set; training the open information extraction model using the first data set and the target data set in a denoising training manner.

[0224] In addition, the logical instructions in the memory 430 described above can be implemented in the form of a software functional unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium, including a number of instructions to make a computer device (which can be a personal computer, server, or network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0225] In a fourth aspect, the present application further provides a computer program product comprising a computer program, which can be stored on a non-transitory computer-readable storage medium, and the computer program can be executed by a processor to enable a computer to perform the training method of the open information extraction model provided by the above-mentioned methods, and the method comprises: obtaining a target data set with natural language sentences as samples; generating a paraphrase of each natural language sentence in the target data set; performing structured knowledge recovery on the paraphrase of each natural language sentence in the target data set to obtain the structured knowledge corresponding to each natural language sentence in the target data set; constructing a first data set with the paraphrases and the structured knowledge corresponding to all natural language sentences in the target data set; and training the open information extraction model by using the first data set and the target data set in a denoising training manner.

[0226] In a fifth aspect, the present application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and the computer program can be executed by a processor to implement the training method of the open information extraction model provided by the above-mentioned methods, and the method comprises: obtaining a target data set with natural language sentences as samples; generating a paraphrase of each natural language sentence in the target data set; performing structured knowledge recovery on the paraphrase of each natural language sentence in the target data set to obtain the structured knowledge corresponding to each natural language sentence in the target data set; constructing a first data set with the paraphrases and the structured knowledge corresponding to all natural language sentences in the target data set; and training the open information extraction model by using the first data set and the target data set in a denoising training manner.

[0227] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. According to actual needs, part or all of the modules can be selected to achieve the purpose of the present embodiment scheme. Those skilled in the art can understand and implement it without creative labor.

[0228] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0229] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.< / sep> < / sep> < / sep>

Claims

1. A training method for an open information extraction model, characterized in that, The method comprises: acquiring a target data set with natural language sentences as samples; generating a paraphrase of each natural language sentence in the target data set; recovering structured knowledge of the paraphrase of each natural language sentence in the target data set to obtain structured knowledge corresponding to each natural language sentence in the target data set; the structured knowledge recovery of the paraphrase of each natural language sentence in the target data set comprises: recovering arguments of the paraphrase of each natural language sentence in the target data set by using a semantic matching method to obtain a first sentence; recovering arguments of the first sentence by using a syntax tree walking method to obtain a second sentence; recovering predicates of the second sentence by using a predicate prediction method to obtain structured knowledge corresponding to the natural language sentence; constructing a first data set with the paraphrase and structured knowledge corresponding to all natural language sentences in the target data set; training an open information extraction model by using a noise reduction training method with the first data set and the target data set; the noise reduction training method comprises: word importance score labeling and argument random masking of the paraphrase in the first data set to obtain a second data set; argument random masking of the target data set; taking the union of the argument random masked target data set and the second data set as an expansion set; training an open information extraction model with the expansion set according to a training target; wherein the training target is expressed as follows: ； In the above formula, are to be extracted triplets, is a sentence in the expanded set, is a sample distribution on the expanded set, is a neural network model.

2. The method of claim 1, wherein, the generation of the paraphrase of each natural language sentence in the target data set comprises: for each natural language sentence in the target data set, inputting a sequence formed by the natural language sentence, a syntax tree corresponding to the natural language sentence and a target syntax tree into a syntax controllable paraphrase generation model to obtain a sentence output by the paraphrase generation model, the sentence complying with the target syntax tree; taking the sentence as the paraphrase of the natural language sentence; wherein the paraphrase generation model is trained by using a parallel corpus ParaNMT-small; each sample in the parallel corpus ParaNMT-small is composed of a natural language sentence and a corresponding paraphrase.

3. The method of claim 2, wherein, the syntax tree corresponding to the natural language sentence is determined in the following manner: performing syntax analysis and bracket tree form representation on the natural language sentence to obtain a syntax tree corresponding to the natural language sentence; the target syntax tree is determined in the following manner: determining the syntax tree corresponding to the natural language sentence and the syntax tree corresponding to the paraphrase in each sample of the parallel corpus ParaNMT-small; The syntax tree pair corresponding to each sample of the parallel corpus ParaNMT-small is generated by truncating the syntax tree corresponding to the natural language sentence and the syntax tree corresponding to the paraphrase sentence in each sample of the parallel corpus ParaNMT-small to only retain the first three layers; The syntax tree pair set is formed by the syntax tree pairs corresponding to all samples of the parallel corpus ParaNMT-small, the first syntax tree set is formed by the syntax trees corresponding to the natural language sentences in all samples of the parallel corpus ParaNMT-small, and the second syntax tree set is formed by the syntax trees corresponding to the paraphrase sentences in all samples of the parallel corpus ParaNMT-small; M syntax trees with the highest similarity to the syntax tree corresponding to the natural language sentence are selected from the first syntax tree set, and the M syntax trees are used as source syntax trees; The distribution function corresponding to each source syntax tree is determined according to the frequency of the syntax tree pair formed by each source syntax tree and each syntax tree in the second syntax tree set in the syntax tree pair set; K syntax trees are sampled from the distribution function corresponding to each source syntax tree to obtain M K syntax trees; The M K syntax trees are all as the target syntax tree; wherein each source syntax tree corresponds to a distribution function As shown below: ； In the above formula, is the source syntax tree, is a syntax tree in the second syntax tree set, is the i-th syntax tree in the second syntax tree set, is the i-th syntax tree in the second syntax tree set, is is the frequency of occurrence of the syntax tree pair in the set of syntax tree pairs, is is the frequency of occurrence of the syntax tree pair in the set of syntax tree pairs.

4. The method of claim 3, wherein, The semantic matching method is used to recover the arguments of the paraphrase sentence to obtain a first sentence, including: The first vectorized embedding representation corresponding to the natural language sentence and the second vectorized embedding representation corresponding to the paraphrase sentence are obtained. For each triple in the natural language sentence, the cosine similarity between the arguments / predicates in the triple and the words in the paraphrase sentence is calculated according to the first vectorized embedding representation and the second vectorized embedding representation to obtain a similarity score sequence corresponding to the arguments / predicates in the triple. The position interval in which the similarity scores in the similarity score sequence corresponding to the arguments / predicates in the triple are successively greater than a preset threshold is used as the candidate position of the arguments / predicates in the triple. The position of the triple is located according to the candidate positions of the arguments in the triple and the candidate positions of the predicates in the triple, with the highest overall similarity and no position overlap as the basis. The first sentence is generated according to the positions of each triple in the natural language sentence.

5. The method of claim 4, wherein, the argument in the triple the corresponding sequence of similarity scores is calculated as follows: ； the predicate in the triple the corresponding sequence of similarity scores , calculated as follows: ; wherein, is a portion of the first vectorized embedding representation corresponding to a first word of the natural language statement, is a portion of the first vectorized embedding representation corresponding to a second word of the natural language statement, is the second vectorized embedding representation, is a first word of the argument, is an ordering index of the first word in the natural language statement, is a last word of the argument, is an ordering index of the last word in the natural language statement, is a first word of the predicate, is an ordering index of the first word in the natural language statement, is a last word of the predicate, is an ordering index of the last word in the natural language statement. 6.The method of Claim 4, wherein, The syntax tree walk method is used to recover the arguments of the first sentence to obtain a second sentence, including: For each word of each argument in the first sentence, the target syntax tree is postorder traversed to find a subtree with NP, QP or NX as the root and containing the word, and the subtree is recorded as the subtree corresponding to the word. All the sub-trees corresponding to the words in the argument are included in a third set. The sequence of words corresponding to each subtree in the third set is used as a replacement option for the argument to generate a replacement set of the argument. The arguments in the first sentence are optimized based on the replacement set of all arguments in the first sentence, with the highest overall similarity and no position overlap as the basis, to obtain the second sentence. 7.The method of Claim 3, wherein, The predicate prediction method is used to recover the predicates of the second sentence to obtain the structured knowledge corresponding to the natural language sentence, including: inputting the second sentence and all arguments in the second sentence into a pre-constructed T5 model to obtain a predicate output by the T5 model; and optimizing the second sentence by using the predicate to obtain structured knowledge corresponding to the natural language sentence; The T5 model is trained by using a predicate prediction data set. The predicate prediction data set is constructed on the basis of the target data set and takes a natural language sentence, all arguments in the natural language sentence, and a predicate in the natural language sentence as sample data.

8. A training device for an open information extraction model, characterized in that, The device comprises: an acquisition module configured to acquire a target data set taking a natural language sentence as sample data; a paraphrase generation module configured to generate a paraphrase of each natural language sentence in the target data set; a structured knowledge recovery module configured to perform structured knowledge recovery on the paraphrase of each natural language sentence in the target data set to obtain structured knowledge corresponding to each natural language sentence in the target data set; the structured knowledge recovery on the paraphrase of each natural language sentence in the target data set to obtain structured knowledge corresponding to each natural language sentence in the target data set is specifically configured to perform argument recovery on the paraphrase of each natural language sentence in the target data set by using a semantic matching manner to obtain a first sentence; perform argument recovery on the first sentence by using a syntax tree walk manner to obtain a second sentence; perform predicate recovery on the second sentence by using a predicate prediction manner to obtain structured knowledge corresponding to the natural language sentence; a first data set generation module configured to construct a first data set by using the paraphrase and the structured knowledge corresponding to all natural language sentences in the target data set; a noise reduction training module configured to train an open information extraction model by using the first data set and the target data set and by using a noise reduction training manner; the noise reduction training of the open information extraction model by using the first data set and the target data set and by using the noise reduction training manner is specifically configured to mark a word importance score of the paraphrase in the first data set and perform argument random masking to obtain a second data set; perform argument random masking on the target data set; take a union of the target data set subjected to argument random masking and the second data set as an expansion set; train an open information extraction model by using the expansion set and according to a training target; The training target expression is as follows: ； In the above formula, are to be extracted triplets, is a sentence in the expanded set, is a sample distribution on the expanded set, is a neural network model.

Images