Event extraction model training method, extraction method, device, equipment and medium

Abstract

This disclosure relates to a training method, device, electronic device, and computer-readable medium for an event extraction model, belonging to the field of natural language processing technology. The training method for the event extraction model includes: acquiring training data from training sample documents, including trigger words, arguments, and irrelevant entities in each event; inputting the training data into a language model to obtain trigger word vectors, argument vectors, and irrelevant entity vectors; obtaining positive sample pairs based on trigger word vectors and argument vectors in the same event, and obtaining negative sample pairs based on irrelevant entity vectors and trigger word vectors and argument vectors in different events; comparing the sample similarity of the positive and negative sample pairs with the sample labels to calculate the model loss; training the model based on the model loss to obtain a pre-trained language model, and further obtaining the event extraction model based on the pre-trained language model. This disclosure improves the performance of the event extraction model by calculating the similarity of positive and negative sample pairs.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of natural language processing, in particular, to an event extraction model training method, an event extraction method, an event extraction model training device, an event extraction device, an electronic device and a computer readable medium. BACKGROUND

[0002] Event extraction is an important part of information extraction, aiming to find the trigger words and corresponding arguments of events from text. Event extraction has wide applications in many fields, such as information retrieval, text summarization, knowledge base construction, public opinion analysis, etc. Event extraction mainly includes the following steps: 1, detection of event trigger words and event types; 2, extraction of event arguments related to the event.

[0003] The pre-trained language model can effectively improve the performance of the event extraction model, and its training includes two steps of pre-training and fine-tuning. The current pre-training method does not involve modeling and learning of events, only modeling the semantic information of events in the fine-tuning stage, and lacks pre-training for the event extraction task, so that the event extraction model cannot fully utilize the information in the data by using the pre-training method, thereby affecting the performance of the model.

[0004] In view of this, there is an urgent need in the art for an event extraction model training method that can fully utilize event information in data in the pre-training stage and improve the performance of the event extraction model.

[0005] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0006] The purpose of the present disclosure is to provide an event extraction model training method, an event extraction method, an event extraction model training device, an event extraction device, an electronic device and a computer readable medium, which can at least to some extent fully utilize event information in data in the pre-training stage and improve the performance of the event extraction model.

[0007] According to a first aspect of the present disclosure, an event extraction model training method is provided, comprising:

[0008] obtaining training data from a training sample document, the training data comprising trigger words and arguments in each event of the training sample document, and irrelevant entities in the training sample document; wherein the irrelevant entities are entities in the training sample document that do not belong to any event;

[0009] The training data is input into the language model to be trained to obtain the corresponding trigger word vectors, argument vectors, and irrelevant entity vectors, respectively.

[0010] Positive sample pairs are obtained based on the trigger word vectors and argument vectors in the same event, and negative sample pairs are obtained based on the irrelevant entity vectors and the trigger word vectors and argument vectors in different events.

[0011] The sample similarity of each positive sample pair is compared with the positive sample label, and the sample similarity of each negative sample pair is compared with the negative sample label. The model loss is calculated based on the similarity comparison results of each positive sample pair and the negative sample pair.

[0012] The model parameters of the language model are iteratively trained based on the model loss, a pre-trained language model is obtained based on the training results, and the event extraction model is obtained by further training based on the pre-trained language model.

[0013] In one exemplary embodiment of this disclosure, obtaining positive sample pairs based on trigger word vectors and argument vectors in the same event includes:

[0014] The positive sample pairs are obtained by combining the trigger word vector corresponding to the trigger word in each event with the argument vector corresponding to each argument in the same event.

[0015] In one exemplary embodiment of this disclosure, obtaining negative sample pairs based on the irrelevant entity vectors and trigger word vectors and argument vectors in different events includes:

[0016] By combining the trigger word vectors and argument vectors belonging to different events in pairs, negative sample pairs are obtained.

[0017] The irrelevant entity vectors are combined pairwise with other irrelevant entity vectors to obtain negative sample pairs.

[0018] The irrelevant entity vectors are combined pairwise with the trigger word vectors or argument vectors in any event to obtain negative sample pairs.

[0019] In one exemplary embodiment of this disclosure, the sample similarity includes the cosine similarity between the two word vectors in each positive sample pair or negative sample pair.

[0020] In one exemplary embodiment of this disclosure, calculating the model loss based on the similarity comparison results of each of the positive sample pairs and the negative sample pairs includes:

[0021] The positive sample similarity difference is obtained based on the sample similarity of the positive sample pair and the positive sample label, and the negative sample similarity difference is obtained based on the sample similarity of the negative sample pair and the negative sample label.

[0022] The model loss is calculated based on the cross-entropy loss function and the similarity difference between the positive and negative samples.

[0023] In one exemplary embodiment of this disclosure, before further training the event extraction model based on the pre-trained language model to obtain the model, the method further includes:

[0024] A first language prediction model is obtained based on the pre-trained language model, and a portion of target words are selected from the training sample documents according to a preset ratio as masking task objects.

[0025] The target words are replaced with the original target words, masked replacement words, and randomly replaced words according to the preset replacement ratio;

[0026] The training sample document after replacing the target word is input into the first language prediction model to obtain the predicted value corresponding to the masking task object, and the first model loss is obtained based on the difference between the predicted value and the true value corresponding to the masking task object.

[0027] The first language prediction model is iteratively trained based on the first model loss, and the pre-trained language model is updated based on the training results of the first language prediction model.

[0028] In one exemplary embodiment of this disclosure, after updating the pre-trained language model based on the training result of the first language prediction model, the method further includes:

[0029] A second language prediction model is obtained based on the pre-trained language model, and a portion of target sentences are selected from the training sample documents as the first sentences. The first sentences are divided into the first positive sample sentences and the first negative sample sentences according to the preset positive and negative sample ratio.

[0030] Obtain the next sentence of the first positive sample statement as the corresponding second positive sample statement, and obtain a positive sample statement pair based on the first positive sample statement and the corresponding second positive sample statement;

[0031] Obtain a random statement from the training sample document as the second negative sample statement corresponding to the first negative sample statement, and obtain a negative sample statement pair based on the first negative sample statement and the corresponding second negative sample statement;

[0032] The positive sample sentence pairs and the negative sample sentence pairs are input into the second language prediction model to obtain the next sentence prediction result corresponding to each positive and negative sample sentence pair, and the second model loss is obtained based on the next sentence prediction result.

[0033] The second language prediction model is iteratively trained based on the second model loss, and the pre-trained language model is updated again based on the training results of the second language prediction model.

[0034] According to a second aspect of this disclosure, an event extraction method is provided, comprising:

[0035] The text content in the target document is input into the event extraction model, wherein the event extraction model is obtained by the training method of the event extraction model as described in any of the above.

[0036] The event extraction results of the target document are obtained based on the output of the event extraction model.

[0037] According to a third aspect of this disclosure, a training apparatus for an event extraction model is provided, comprising:

[0038] The training data acquisition module is used to acquire training data from training sample documents. The training data includes trigger words and arguments in each event of the training sample documents, as well as irrelevant entities in the training sample documents. The irrelevant entities are entities in the training sample documents that do not belong to any event.

[0039] The word vector output module is used to input the training data into the language model to be trained, and obtain the corresponding trigger word vectors, argument vectors and irrelevant entity vectors respectively;

[0040] The sample pair generation module is used to generate positive sample pairs based on the trigger word vectors and argument vectors in the same event, and to generate negative sample pairs based on the irrelevant entity vectors and the trigger word vectors and argument vectors in different events.

[0041] The model loss calculation module is used to compare the sample similarity of each positive sample pair with the positive sample label, compare the sample similarity of each negative sample pair with the negative sample label, and calculate the model loss based on the similarity comparison results of each positive sample pair and the negative sample pair.

[0042] The model parameter training module is used to iteratively train the model parameters of the language model based on the model loss, obtain a pre-trained language model based on the training results, and further train the event extraction model based on the pre-trained language model.

[0043] According to a fourth aspect of this disclosure, a training apparatus for an event extraction model is provided, comprising:

[0044] A text content input module is used to input the text content of the target document into the event extraction model, wherein the event extraction model is obtained by the training method of the event extraction model described above.

[0045] The extraction result determination module is used to obtain the event extraction result of the target document based on the output result of the event extraction model.

[0046] According to a fifth aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform a training method for an event extraction model as described in any of the preceding claims by executing the executable instructions.

[0047] According to a sixth aspect of this disclosure, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the training method for the event extraction model described in any of the preceding claims.

[0048] The exemplary embodiments disclosed herein can have the following beneficial effects:

[0049] In the training method of the event extraction model according to the exemplary implementation of this disclosure, on the one hand, positive and negative sample pairs are constructed based on trigger words and arguments of the same event in the training sample documents, as well as irrelevant entities that do not belong to any event. This automatically generates data that can be used for pre-training of contrastive learning, explicitly modeling the differences between different events from the data, thereby making fuller use of the event information in the data. This reduces the workload of manual annotation while improving the performance of the subsequent event extraction model. On the other hand, by calculating the similarity between positive and negative sample pairs and completing the pre-training of the model based on contrastive learning, the semantic representation of the event is learned, making the representation of words in the same event closer to the representation of unrelated words, thereby improving the performance of the event extraction model and the effectiveness of the event extraction task. The training method of the event extraction model in the exemplary implementation of this disclosure is not targeted at a specific event extraction model and can be applied to all event extraction based on pre-trained models, thus having a wide range of application prospects.

[0050] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0051] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0052] Figure 1 A flowchart illustrating the training method of the event extraction model according to an exemplary embodiment of the present disclosure is shown.

[0053] Figure 2 A schematic diagram illustrating the construction of positive and negative sample pairs according to a specific embodiment of the present disclosure is shown;

[0054] Figure 3 A schematic diagram illustrating the process of pre-training a model based on a next-sentence prediction task in an exemplary embodiment of this disclosure is shown.

[0055] Figure 4 A schematic diagram of the process for pre-training a model based on a masking task is shown in an exemplary embodiment of this disclosure;

[0056] Figure 5 A flowchart illustrating an example embodiment of the event extraction method of this disclosure is shown.

[0057] Figure 6 A flowchart illustrating a training method for an event extraction model according to a specific embodiment of the present disclosure is shown.

[0058] Figure 7 A block diagram of a training apparatus for an event extraction model according to an exemplary embodiment of the present disclosure is shown;

[0059] Figure 8 A block diagram of an event extraction apparatus according to an exemplary embodiment of the present disclosure is shown;

[0060] Figure 9 A schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure is shown. Detailed Implementation

[0061] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0062] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0063] This example implementation first provides a method for training an event extraction model. (See reference...) Figure 1 As shown, the training method for the above event extraction model may include the following steps:

[0064] Step S110. Obtain training data from the training sample documents. The training data includes trigger words and arguments in each event of the training sample documents, as well as irrelevant entities in the training sample documents.

[0065] Irrelevant entities are entities in the training sample documents that do not belong to any event.

[0066] Step S120. Input the training data into the language model to be trained to obtain the corresponding trigger word vectors, argument vectors and irrelevant entity vectors.

[0067] Step S130. Obtain positive sample pairs based on the trigger word vectors and argument vectors in the same event, and obtain negative sample pairs based on irrelevant entity vectors and trigger word vectors and argument vectors in different events.

[0068] Step S140. Compare the sample similarity of each positive sample pair with the positive sample label, compare the sample similarity of each negative sample pair with the negative sample label, and calculate the model loss based on the similarity comparison results of each positive and negative sample pair.

[0069] Step S150. Iteratively train the model parameters of the language model according to the model loss, obtain a pre-trained language model based on the training results, and further train the event extraction model based on the pre-trained language model.

[0070] In the training method of the event extraction model according to the exemplary implementation of this disclosure, on the one hand, positive and negative sample pairs are constructed based on trigger words and arguments of the same event in the training sample documents, as well as irrelevant entities that do not belong to any event. This automatically generates data that can be used for pre-training of contrastive learning, explicitly modeling the differences between different events from the data, thereby making fuller use of the event information in the data. This reduces the workload of manual annotation while improving the performance of the subsequent event extraction model. On the other hand, by calculating the similarity between positive and negative sample pairs and completing the pre-training of the model based on contrastive learning, the semantic representation of the event is learned, making the representation of words in the same event closer to the representation of unrelated words, thereby improving the performance of the event extraction model and the effectiveness of the event extraction task. The training method of the event extraction model in the exemplary implementation of this disclosure is not targeted at a specific event extraction model and can be applied to all event extraction based on pre-trained models, thus having a wide range of application prospects.

[0071] Below, in conjunction with Figures 2 to 4 The steps described above in this example implementation will be explained in more detail.

[0072] In step S110, training data is obtained from the training sample documents. The training data includes trigger words and arguments in each event of the training sample documents, as well as irrelevant entities in the training sample documents.

[0073] A document contains multiple different events, each including a trigger word and multiple arguments. The trigger word represents the core word of the event, usually a verb or noun. Arguments represent the participants in the event, mainly composed of entities, values, and timestamps. Values ​​are non-entity participants, such as job titles. Irrelevant entities are entities in the training sample documents that do not belong to any event.

[0074] In this example implementation, for each event in the training sample document, the trigger words and arguments are obtained, then all entities in the document are obtained, and irrelevant entities that do not belong to any event are filtered out. Training data is then obtained based on these data.

[0075] In step S120, the training data is input into the language model to be trained, and the corresponding trigger word vectors, argument vectors and irrelevant entity vectors are obtained respectively.

[0076] In this example implementation, the language model is used to obtain word vectors corresponding to the text. Training data is input into the language model to be trained to obtain word vectors corresponding to the trigger words and arguments in each event, as well as word vectors for all irrelevant entities in the document.

[0077] In step S130, positive sample pairs are obtained based on the trigger word vectors and argument vectors in the same event, and negative sample pairs are obtained based on irrelevant entity vectors and trigger word vectors and argument vectors in different events.

[0078] In this example implementation, positive sample pairs can be obtained based on the trigger word vectors and argument vectors of the same event. Specifically, the trigger word vector corresponding to the trigger word in each event can be combined with the argument vectors corresponding to each argument in the same event to obtain positive sample pairs. For example, event 1 contains trigger word 1 and argument 1 and argument 2. By combining the word vector of trigger word 1 with the word vectors of argument 1 and argument 2 respectively, two positive sample pairs can be constructed.

[0079] In this example implementation, negative sample pairs can be obtained based on irrelevant entity vectors and trigger word vectors and argument vectors in different events. The specific methods for constructing negative sample pairs can include the following three:

[0080] The first method involves pairwise combining the trigger word vectors and argument vectors belonging to different events to obtain negative sample pairs. For example, the word vector of trigger word 1 in event 1 can be combined with the word vector of trigger word 2 in event 2 to construct a negative sample pair; or the word vector of trigger word 1 in event 1 can be combined with the word vector of any argument in event 2 to construct a negative sample pair; or the word vector of any argument in event 1 can be combined with the word vector of any argument in event 2 to construct a negative sample pair.

[0081] The second method involves pairwise combining irrelevant entity vectors with other irrelevant entity vectors to obtain negative sample pairs. Combining the word vectors of any two irrelevant entities that do not belong to any event can also construct negative sample pairs.

[0082] The third method involves pairwise combining irrelevant entity vectors with trigger word vectors or argument vectors from any event to obtain negative sample pairs. Negative sample pairs can also be constructed by combining word vectors of irrelevant entities that do not belong to any event with word vectors of trigger words or arguments that belong to a particular event.

[0083] like Figure 2The diagram shown is a schematic diagram of constructing positive and negative sample pairs according to a specific embodiment of the present disclosure, and provides exemplary methods for constructing positive sample pairs based on two different events and constructing negative sample pairs according to the above three methods.

[0084] In step S140, the sample similarity of each positive sample pair is compared with the positive sample label, and the sample similarity of each negative sample pair is compared with the negative sample label. The model loss is calculated based on the similarity comparison results of each positive and negative sample pair.

[0085] In this example implementation, the sample similarity can be the cosine similarity between two word vectors in each positive or negative sample pair, i.e., the cosine similarity between two word vectors in a positive sample pair and the cosine similarity between two word vectors in a negative sample pair. In addition, other similarity calculation methods can be used to obtain the sample similarity between two word vectors in a sample pair; this example implementation does not impose any specific limitations.

[0086] In this example implementation, the label of the positive sample pair can be set to 1, and the label of the negative sample pair can be set to 0. The sample similarity of each positive sample pair is compared with the positive sample label, and the sample similarity of each negative sample pair is compared with the negative sample label. Then, the positive sample similarity difference is obtained based on the sample similarity of the positive sample pair and the positive sample label, and the negative sample similarity difference is obtained based on the sample similarity of the negative sample pair and the negative sample label. In this way, the positive sample pairs and negative sample pairs are subjected to comparative learning pre-training.

[0087] Contrastive learning focuses on learning the common characteristics among similar instances and distinguishing the differences between instances of different classes. It doesn't need to focus on the details of instances; it only needs to learn to differentiate data at the abstract semantic level of the feature space, thus enhancing the model's generalization ability. The ultimate goal of contrastive learning is to learn an encoder that can produce similar encodings for data of similar instances and differentiate the encodings for data of different classes.

[0088] The text encoder uses self-supervised contrastive learning to learn semantic representations of events, thereby making the representations of words in the same event closer to those of other unrelated words, thus improving the performance of the event extraction model.

[0089] After obtaining the similarity comparison results of positive and negative sample pairs, the model loss can be calculated based on the cross-entropy loss function and the similarity difference between positive and negative samples.

[0090] In step S150, the model parameters of the language model are iteratively trained according to the model loss, a pre-trained language model is obtained based on the training results, and the event extraction model is obtained by further training based on the pre-trained language model.

[0091] In this example implementation, gradient descent can be used to update parameters, train the language model, and obtain a pre-trained language model based on the training results. Then, based on the pre-trained language model, a neural network is combined for further training and adjustment to obtain the final event extraction model.

[0092] In this example implementation, before further training the event extraction model based on the pre-trained language model, the model can also be pre-trained based on the next sentence prediction task and the masking task. For example... Figure 3 As shown, the method for pre-training a model based on the next sentence prediction task can specifically include the following steps:

[0093] Step S310. Obtain the first language prediction model based on the pre-trained language model, and select a portion of target words from the training sample documents as masking task objects according to a preset ratio.

[0094] In this example implementation, a classification layer can be added to the language model to obtain a first language prediction model. Then, a portion of target words can be extracted from the original training sample documents as objects for the masking task. For example, 15% of the tokens (words) can be randomly selected as objects to participate in the masking task.

[0095] Step S320. Replace the target words with the original target words, masked replacement words, and randomly replaced words according to the preset replacement ratio.

[0096] The target word can be divided into three parts according to a certain ratio, and each part can be replaced in different ways to obtain the original target word, the masked replacement word, and the random replacement word. For example, with an 80% probability, the token is replaced with the [MASK] (mask) tag; with a 10% probability, the token is replaced with a random word; and with a 10% probability, the token is left unchanged.

[0097] Step S330. Input the training sample document after replacing the target word into the first language prediction model to obtain the predicted value corresponding to the masked task object, and obtain the first model loss based on the difference between the predicted value and the true value corresponding to the masked task object.

[0098] After replacing the target words, the information is input into the first language prediction model. The model is then used to predict the masked tokens, and the loss of the first model is calculated based on the difference between the predicted and the true values.

[0099] Step S340. Iteratively train the first language prediction model based on the first model loss, and update the pre-trained language model based on the training results of the first language prediction model.

[0100] The model parameters in the first language prediction model are iteratively updated based on the first model loss. After training is completed, the language model is extracted to obtain the updated pre-trained language model.

[0101] In this example implementation, as Figure 4 As shown, the method for pre-training a model based on the masking task can specifically include the following steps:

[0102] Step S410. Obtain the second language prediction model based on the pre-trained language model, and select some target sentences from the training sample documents as the first sentences. Divide the first sentences into the first positive sample sentences and the first negative sample sentences according to the preset positive and negative sample ratio.

[0103] In this example implementation, a sentence classification layer can be added to the language model to obtain a second language prediction model. Then, a portion of the target sentences can be extracted from the original training sample documents as the first sentences, and the first sentences participating in the training can be denoted as sentence A.

[0104] Based on a preset positive and negative sample ratio, such as 50%:50%, the first statement is divided into the first positive sample statement and the first negative sample statement.

[0105] Step S420. Obtain the next sentence of the first positive sample statement as the corresponding second positive sample statement, and obtain the positive sample statement pair based on the first positive sample statement and the corresponding second positive sample statement.

[0106] For example, with a 50% probability, the sentence following sentence A in the original text is used as sentence B, and sentence A and sentence B are combined to construct a positive sample sentence pair.

[0107] Step S430. Obtain a random statement from the training sample document as the second negative sample statement corresponding to the first negative sample statement, and obtain a negative sample statement pair based on the first negative sample statement and the corresponding second negative sample statement.

[0108] For example, with an additional 50% probability, a randomly selected sentence is used as sentence B, and sentence A and sentence B are combined to construct a negative sample sentence pair.

[0109] Step S440. Input the positive sample sentence pairs and negative sample sentence pairs into the second language prediction model to obtain the next sentence prediction result corresponding to each positive and negative sample sentence pair, and obtain the second model loss based on the next sentence prediction result.

[0110] Positive and negative sample sentence pairs are input into a second language prediction model, which is then used to predict the relationship between sentence A and sentence B. The second language prediction model outputs the probability that sentence B is the next sentence of sentence A, and the second model loss is calculated based on this probability value.

[0111] Step S450. Iteratively train the second language prediction model based on the second model loss, and update the pre-trained language model again based on the training results of the second language prediction model.

[0112] The model parameters in the second language prediction model are iteratively updated based on the second model loss. After training is completed, the language model is extracted to obtain the updated pre-trained language model.

[0113] Updating the model parameters of the pre-trained language model based on the next sentence prediction task and the masking task can further improve the performance of the subsequent event extraction model.

[0114] Building upon this, this example implementation also provides an event extraction method. (See reference) Figure 5 As shown, the training method for the above event extraction model may include the following steps:

[0115] Step S510. Input the text content from the target document into the event extraction model.

[0116] The event extraction model is obtained through the training method described above.

[0117] Step S520. Obtain the event extraction results of the target document based on the output of the event extraction model.

[0118] Event extraction models can extract events that users are interested in from target documents, as well as keywords in those events, including trigger words and arguments, and finally present them to users in a structured manner.

[0119] like Figure 6 The diagram shown is a complete flowchart of the training method for the event extraction model in a specific embodiment of this disclosure. It illustrates the steps described above in this example embodiment. The specific steps of the flowchart are as follows:

[0120] Step S610. Input a training sample document.

[0121] Step S620. Combine trigger words and arguments belonging to the same event to form positive sample pairs.

[0122] For each event in the document, the trigger words and arguments belonging to the same event are combined in pairs to form positive samples.

[0123] Step S630. Construct negative sample pairs for trigger words and arguments that do not belong to the same event and for entities that do not belong to any event.

[0124] Negative samples are constructed for trigger words and arguments that do not belong to the same event, and for entities that do not belong to any event.

[0125] Step S640. Calculate the similarity between positive sample pairs and negative sample pairs.

[0126] Calculate the cosine similarity between two word vectors in a positive sample pair, and the cosine similarity between two word vectors in a negative sample pair.

[0127] Step S650. Set the label of positive sample pairs to 1 and the label of negative sample pairs to 0, and perform contrastive learning pre-training.

[0128] The sample similarity of each positive sample pair is compared with the positive sample label 1, and the sample similarity of each negative sample pair is compared with the negative sample label 0, for comparative learning pre-training.

[0129] Step S660. Pre-train the model on the "next sentence prediction" and "mask language model" tasks.

[0130] The model is pre-trained based on the next sentence prediction task and the masking task to further improve the performance of the subsequent event extraction model.

[0131] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0132] Furthermore, this disclosure also provides a training apparatus for an event extraction model. (See reference) Figure 7 As shown, the training device for this event extraction model may include a training data acquisition module 710, a word vector output module 720, a sample pair generation module 730, a model loss calculation module 740, and a model parameter training module 750. Wherein:

[0133] The training data acquisition module 710 can be used to acquire training data from training sample documents. The training data includes trigger words and arguments in each event of the training sample documents, as well as irrelevant entities in the training sample documents; wherein, irrelevant entities are entities in the training sample documents that do not belong to any event.

[0134] The word vector output module 720 can be used to input training data into the language model to be trained, and obtain the corresponding trigger word vectors, argument vectors and irrelevant entity vectors respectively;

[0135] The sample pair generation module 730 can be used to generate positive sample pairs based on trigger word vectors and argument vectors in the same event, and to generate negative sample pairs based on irrelevant entity vectors and trigger word vectors and argument vectors in different events.

[0136] The model loss calculation module 740 can be used to compare the sample similarity of each positive sample pair with the positive sample label, compare the sample similarity of each negative sample pair with the negative sample label, and calculate the model loss based on the similarity comparison results of each positive sample pair and negative sample pair.

[0137] The model parameter training module 750 can be used to iteratively train the model parameters of the language model based on the model loss, obtain a pre-trained language model based on the training results, and further train the event extraction model based on the pre-trained language model.

[0138] In some exemplary embodiments of this disclosure, the sample pair generation module 730 may include a positive sample pair generation unit, which can be used to combine the trigger word vector corresponding to the trigger word in each event with the argument vector corresponding to each argument in the same event to obtain a positive sample pair.

[0139] In some exemplary embodiments of this disclosure, the sample pair generation module 730 may further include a first negative sample pair generation unit, a second negative sample pair generation unit, and a third negative sample pair generation unit.

[0140] in:

[0141] The first negative sample pair generation unit can be used to combine the trigger word vectors and argument vectors belonging to different events in pairs to obtain negative sample pairs.

[0142] The second negative sample pair generation unit can be used to combine irrelevant entity vectors with other irrelevant entity vectors in pairs to obtain negative sample pairs.

[0143] The third negative sample pair generation unit can be used to combine irrelevant entity vectors with trigger word vectors or argument vectors in any event in pairs to obtain negative sample pairs.

[0144] In some exemplary embodiments of this disclosure, the model loss calculation module 740 may include a similarity comparison unit and a cross-entropy loss calculation unit. Wherein:

[0145] The similarity comparison unit can be used to obtain the difference in positive sample similarity based on the sample similarity of positive sample pairs and the positive sample label, and to obtain the difference in negative sample similarity based on the sample similarity of negative sample pairs and the negative sample label.

[0146] The cross-entropy loss calculation unit can be used to calculate the model loss based on the cross-entropy loss function and the similarity difference between positive and negative samples.

[0147] In some exemplary embodiments of this disclosure, the training apparatus for an event extraction model provided in this disclosure may further include a mask task training module, which may include a target word acquisition unit, a target word replacement unit, a first loss calculation unit, and a first model training unit. Wherein:

[0148] The target word acquisition unit can be used to obtain a first language prediction model based on a pre-trained language model, and select a portion of target words from the training sample documents as masking task objects according to a preset ratio.

[0149] The target word replacement unit can be used to replace target words with the original target words, masked replacement words, and random replacement words according to a preset replacement ratio;

[0150] The first loss calculation unit can be used to input the training sample document after replacing the target word into the first language prediction model to obtain the predicted value corresponding to the masked task object, and obtain the first model loss based on the difference between the predicted value and the true value corresponding to the masked task object.

[0151] The first model training unit can be used to iteratively train the first language prediction model based on the first model loss, and update the pre-trained language model based on the training results of the first language prediction model.

[0152] In some exemplary embodiments of this disclosure, the training apparatus for an event extraction model provided in this disclosure may further include a next-sentence prediction training module. This next-sentence prediction training module may include a first sentence acquisition unit, a positive sample sentence pair construction unit, a negative sample sentence pair construction unit, a second loss calculation unit, and a second model training unit. Wherein:

[0153] The first sentence acquisition unit can be used to obtain a second language prediction model based on a pre-trained language model, and select some target sentences from the training sample documents as the first sentence, and divide the first sentence into the first positive sample sentence and the first negative sample sentence according to the preset positive and negative sample ratio.

[0154] The positive sample statement pair construction unit can be used to obtain the next sentence of the first positive sample statement as the corresponding second positive sample statement, and to obtain the positive sample statement pair based on the first positive sample statement and the corresponding second positive sample statement.

[0155] The negative sample statement pair construction unit can be used to obtain random statements in the training sample document as the second negative sample statement corresponding to the first negative sample statement, and obtain negative sample statement pairs based on the first negative sample statement and the corresponding second negative sample statement.

[0156] The second loss calculation unit can be used to input positive sample sentence pairs and negative sample sentence pairs into the second language prediction model, obtain the next sentence prediction result corresponding to each positive and negative sample sentence pair, and obtain the second model loss based on the next sentence prediction result.

[0157] The second model training unit can be used to iteratively train the second language prediction model based on the second model loss, and update the pre-trained language model again based on the training results of the second language prediction model.

[0158] Furthermore, this disclosure also provides an event extraction apparatus. (See reference...) Figure 8 As shown, the event extraction device may include a text content input module 810 and an extraction result determination module 820.

[0159] in:

[0160] The text content input module 810 can be used to input the text content in the target document into the event extraction model, wherein the event extraction model is obtained through the training device of the event extraction model described above.

[0161] The extraction result determination module 820 can be used to obtain the event extraction results of the target document based on the output results of the event extraction model.

[0162] The specific details of the training device and each module / unit in the event extraction device of the above event extraction model have been described in detail in the corresponding method implementation section, and will not be repeated here.

[0163] Figure 9 A schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure is shown.

[0164] It should be noted that, Figure 9 The computer system 900 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0165] like Figure 9As shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 902 or programs loaded from storage section 908 into random access memory (RAM) 903. The RAM 903 also stores various programs and data required for system operation. The CPU 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0166] The following components are connected to I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to I / O interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 910 as needed so that computer programs read from it can be installed into storage section 908 as needed.

[0167] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 909, and / or installed from removable medium 911. When the computer program is executed by central processing unit (CPU) 901, it performs various functions defined in the system of this disclosure.

[0168] It should be noted that the computer-readable medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0169] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0170] In another aspect, this disclosure also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.

[0171] It should be noted that although several modules for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided and embodied by multiple modules.

[0172] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein.

[0173] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A training method for an event extraction model, characterized in that, include: Training data is obtained from training sample documents, the training data including trigger words and arguments in each event of the training sample documents, and irrelevant entities in the training sample documents; wherein, the irrelevant entities are entities in the training sample documents that do not belong to any event; The training data is input into the language model to be trained to obtain the corresponding trigger word vectors, argument vectors, and irrelevant entity vectors, respectively. Positive sample pairs are obtained based on trigger word vectors and argument vectors from the same event, and negative sample pairs are obtained based on the irrelevant entity vectors and trigger word vectors and argument vectors from different events. Obtaining negative sample pairs based on the irrelevant entity vectors and trigger word vectors and argument vectors from different events includes: pairwise combining trigger word vectors and argument vectors belonging to different events to obtain negative sample pairs; pairwise combining the irrelevant entity vectors with other irrelevant entity vectors to obtain negative sample pairs; and pairwise combining the irrelevant entity vectors with trigger word vectors or argument vectors from any event to obtain negative sample pairs. The sample similarity of each positive sample pair is compared with the positive sample label, and the sample similarity of each negative sample pair is compared with the negative sample label. The model loss is calculated based on the similarity comparison results of each positive sample pair and the negative sample pair. The model parameters of the language model are iteratively trained based on the model loss, a pre-trained language model is obtained based on the training results, and the event extraction model is obtained by further training based on the pre-trained language model.

2. The training method for the event extraction model according to claim 1, characterized in that, The method of obtaining positive sample pairs based on trigger word vectors and argument vectors in the same event includes: The positive sample pairs are obtained by combining the trigger word vector corresponding to the trigger word in each event with the argument vector corresponding to each argument in the same event.

3. The training method for the event extraction model according to claim 1, characterized in that, The sample similarity includes the cosine similarity between the two word vectors in each positive sample pair or negative sample pair.

4. The training method for the event extraction model according to claim 1, characterized in that, The calculation of the model loss based on the similarity comparison results of each positive sample pair and the negative sample pair includes: The positive sample similarity difference is obtained based on the sample similarity of the positive sample pair and the positive sample label, and the negative sample similarity difference is obtained based on the sample similarity of the negative sample pair and the negative sample label. The model loss is calculated based on the cross-entropy loss function and the similarity difference between the positive and negative samples.

5. The training method for the event extraction model according to claim 1, characterized in that, Before further training based on the pre-trained language model to obtain the event extraction model, the method further includes: A first language prediction model is obtained based on the pre-trained language model, and a portion of target words are selected from the training sample documents according to a preset ratio as masking task objects. The target words are replaced with the original target words, masked replacement words, and randomly replaced words according to the preset replacement ratio; The training sample document after replacing the target word is input into the first language prediction model to obtain the predicted value corresponding to the masking task object, and the first model loss is obtained based on the difference between the predicted value and the true value corresponding to the masking task object. The first language prediction model is iteratively trained based on the first model loss, and the pre-trained language model is updated based on the training results of the first language prediction model.

6. The training method for the event extraction model according to claim 5, characterized in that, After updating the pre-trained language model based on the training results of the first language prediction model, the method further includes: A second language prediction model is obtained based on the pre-trained language model, and a portion of target sentences are selected from the training sample documents as the first sentences. The first sentences are divided into the first positive sample sentences and the first negative sample sentences according to the preset positive and negative sample ratio. Obtain the next sentence of the first positive sample statement as the corresponding second positive sample statement, and obtain a positive sample statement pair based on the first positive sample statement and the corresponding second positive sample statement; Obtain a random statement from the training sample document as the second negative sample statement corresponding to the first negative sample statement, and obtain a negative sample statement pair based on the first negative sample statement and the corresponding second negative sample statement; The positive sample sentence pairs and the negative sample sentence pairs are input into the second language prediction model to obtain the next sentence prediction result corresponding to each positive and negative sample sentence pair, and the second model loss is obtained based on the next sentence prediction result. The second language prediction model is iteratively trained based on the second model loss, and the pre-trained language model is updated again based on the training results of the second language prediction model.

7. An event extraction method, characterized in that, include: The text content in the target document is input into the event extraction model, wherein the event extraction model is obtained by the training method of the event extraction model as described in any one of claims 1-6; The event extraction results of the target document are obtained based on the output of the event extraction model.

8. A training device for an event extraction model, characterized in that, include: The training data acquisition module is used to acquire training data from training sample documents. The training data includes trigger words and arguments in each event of the training sample documents, as well as irrelevant entities in the training sample documents. The irrelevant entities are entities in the training sample documents that do not belong to any event. The word vector output module is used to input the training data into the language model to be trained, and obtain the corresponding trigger word vectors, argument vectors and irrelevant entity vectors respectively; The sample pair generation module is used to generate positive sample pairs based on trigger word vectors and argument vectors in the same event, and to generate negative sample pairs based on the irrelevant entity vectors and trigger word vectors and argument vectors in different events. The generation of negative sample pairs based on the irrelevant entity vectors and trigger word vectors and argument vectors in different events includes: pairwise combining trigger word vectors and argument vectors belonging to different events to obtain negative sample pairs; pairwise combining the irrelevant entity vectors with other irrelevant entity vectors to obtain negative sample pairs; and pairwise combining the irrelevant entity vectors with trigger word vectors or argument vectors in any event to obtain negative sample pairs. The model loss calculation module is used to compare the sample similarity of each positive sample pair with the positive sample label, compare the sample similarity of each negative sample pair with the negative sample label, and calculate the model loss based on the similarity comparison results of each positive sample pair and the negative sample pair. The model parameter training module is used to iteratively train the model parameters of the language model based on the model loss, obtain a pre-trained language model based on the training results, and further train the event extraction model based on the pre-trained language model.

9. An event extraction device, characterized in that, include: A text content input module is used to input the text content of the target document into the event extraction model, wherein the event extraction model is obtained by the training method of the event extraction model as described in any one of claims 1-6; The extraction result determination module is used to obtain the event extraction result of the target document based on the output result of the event extraction model.

10. An electronic device, characterized in that, include: processor; as well as A memory for storing one or more programs, which, when executed by the processor, cause the processor to implement the training method of the event extraction model as described in any one of claims 1 to 6 or the event extraction method as described in claim 7.

11. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the training method of the event extraction model as described in any one of claims 1 to 6 or the event extraction method as described in claim 7.

Images