Event argument detection method and system based on label sequence consistency modeling

Abstract

This invention proposes an event argument detection method and system based on label sequence consistency modeling. It mainly includes word sequence semantic encoding, word label sequence annotation, error-prone label sequence generation, and contrastive learning regularization. Word sequence semantic encoding uses BERT and a trained language model to learn semantic representations of preprocessed words, incorporating event type information into the representation vector. Word label sequence annotation uses a fully connected network to predict the probability distribution of the label corresponding to each word. Error-prone label sequence generation generates error-prone label sequences according to a certain strategy based on the probability distribution of the word label sequences. Contrastive learning regularization constructs a regularization loss based on the contrastive learning between error-prone label sequences and correct label sequences, improving the consistency of word sequence labels.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information extraction, in particular, to a method for improving the effect of event argument detection in event extraction tasks. BACKGROUND

[0002] Event, as a structured representation of information, refers to an actual happening involving certain participants. As a special information extraction task, the goal of event extraction is to extract instances of predefined event types from a given text. An event consists of two parts: trigger and argument. The trigger is the word in the text that best expresses the occurrence of the event, usually the core verb of the event sentence; the argument is an entity related to the event and plays a certain role in the event. Generally speaking, event extraction can be divided into two tasks: trigger extraction and argument extraction. The goal of trigger extraction is to find the trigger of the event and determine the event type; the goal of argument extraction is to determine whether an entity in the text is a relevant argument of the event, and if so, to determine the role the entity plays in the event, such as initiator, recipient, attacker, and attacked. Through the advancement of event extraction research in recent years, existing methods have good results for trigger extraction tasks, while in argument extraction, most techniques simplify the candidate argument entities as known information. However, in real-world application scenarios, most cases require complete structured event information to be extracted from pure text, which requires the identification of candidate argument entities from the text first. This step, as a sub-task of argument extraction, is called event argument detection. Existing event argument detection tasks are mostly based on sequence labeling methods, which have some shortcomings:

[0003] 1) In the existing sequence labeling-based event argument detection technology, most of them do not consider the label sequence consistency modeling, and the label sequence consistency here specifically refers to whether the word label sequences within the argument are correct, complete and compatible. The sequence labeling scheme converts the event argument detection task into a problem of labeling all words in the text with corresponding labels, which makes the premise of correctly detecting a certain argument referent that all word labels within it must be labeled correctly. This constraint makes the accuracy requirement for sequence labeling very strict and the global consistency requirement for word sequence labels very high. In addition, the decoding method of taking local optimal label for each word further exacerbates the inconsistency of the final prediction result, and there are various missing error phenomena, including argument internal label mislabeling, isolated label outside the argument, etc. In order to solve this problem, some methods model the transition probability between labels based on conditional random field, but its training needs to calculate the probability of all possible label sequences, and a large number of probability transition calculations are also needed in decoding, which has high computational complexity, and the one-way nature of conditional random field also limits its modeling effect for global consistency, which also makes the final mitigation effect for argument internal label missing and other errors limited.

[0004] 2) Another defect of the sequence labeling-based scheme is that the number distribution of each label is extremely uneven, because the sum of the number of words of all argument referents relative to the number of words in the entire input text still accounts for a low proportion, and the BIO label system will convert the words in the text that are not argument referents into O (Outside, non-argument internal word) label in sequence labeling, which leads to O label will be much more than B (Begin, first word inside the argument) and I (Inside, other words inside the argument). The long-tail distribution of labels makes the model with sequence labeling as a single target prone to overfitting, thereby affecting the accuracy and integrity of the final decoding result. SUMMARY

[0005] In view of the deficiencies of the prior art, the present application proposes an event argument detection method based on label sequence consistency modeling, which comprises:

[0006] A training corpus preprocessing step is used to obtain a training corpus with labeled event argument role categories and event types, perform word segmentation on the text of the training corpus, and obtain the ID of each word in the pre-training dictionary of the language representation model BERT according to the pre-training dictionary of the language representation model BERT;

[0007] A word sequence semantic encoding step is used to input the word sequence composed of all word IDs into the multi-layer translation model Transformer model of BERT to pre-encode the sub-word sequence, map the event type into a distributed expression vector, and respectively splice the distributed expression vector with each word vector and fuse them through a linear network to obtain a word semantic expression vector fused with event type information.

[0008] a word semantic expression vector is input into a full connection network to obtain a probability distribution of each word semantic expression vector belonging to each argument role category, and an argument role category with the highest probability in the probability distribution is selected as a predicted argument role category;

[0009] an error label sequence generation step, according to the predicted argument role category of each word and the annotated argument role category, the word sequence is divided into a correct label sequence predicted correctly and an error label sequence predicted incorrectly;

[0010] a contrast learning regularization step, representation learning is performed on the error label sequence and the correct label sequence, and a loss thereof is taken as a regularization term to train the full connection network and the Transformer model;

[0011] an argument detection step, a text to be detected and an event type thereof are input into the trained Transformer model and the full connection network in sequence to obtain an argument role category of the text.

[0012] The event argument detection method based on label sequence consistency modeling, wherein the word sequence semantic encoding step comprises:

[0013] a BERT language model pre-trained through a corpus is used to pre-encode an input word sequence T to obtain more rich dynamic semantic expressions C={c0, c1, …, c n};

[0014] semantic information of an input event type E is separately encoded, a parameter matrix V is used as an expression vector of each event type to participate in model training, a vector corresponding to the event type is spliced with an output vector of the BERT to obtain an intermediate expression X={x0, x1, …, x n}, information of the two is interactively fused through a full connection network to obtain a final context vector representation H={h0, h1, …, h n} of each word fusion event information, and the overall calculation process is as follows:

[0015] x i =[c i ||V(E)]

[0016] h i =W2·(ReLU(W1·x i +b1))+b2

[0017] V(E) represents the vector representation of event type E in parameter matrix V; || is the vector concatenation operation; W1, W2, b1, b2 are linear transformation matrices and their corresponding bias terms; ReLU is the activation function.

[0018] The event argument detection method based on label sequence consistency modeling, wherein the word label sequence annotation step includes:

[0019] For each word, the context representation of the fusion event information h i Using a linear layer, predict its label probability distribution P = {p0, p1, ..., p n}, p i The vector representing the probability distribution of the label for the i-th word:

[0020] z i =W3·h i +b3

[0021]

[0022] Where W3 and b3 are the linear transformation matrix and the corresponding bias term, respectively, the transformation yields a 3D vector, and j, k belong to {0,1,2}, representing the indices corresponding to labels O, B, and I, respectively. This represents the probability that the i-th word is labeled as the k-th label; the local predicted label corresponding to the input text. in The soft tag corresponding to the i-th word is a vector of length 3.

[0023] Calculate the loss, cross-entropy, and loss function corresponding to the optimized sequence labeling part. As the loss function for sequence labeling tasks:

[0024]

[0025] in This represents the actual label corresponding to the i-th word at the j-th position.

[0026] The event argument detection method based on label sequence consistency modeling, wherein the error-prone label sequence generation step includes:

[0027] Standard correct label sequence converted to in The correct label for the i-th word;

[0028] When the local prediction label L pred The hard tag sequence L is obtained through greedy decoding. greedy When the sequence is inconsistent with the correct label sequence, the sequence is used as a negative sample with an incorrect label, and the negative sample set L is generated.neg = L greedy ; according to the local prediction soft label P obtained by the sequence labeling module, the process of greedy decoding includes:

[0029]

[0030]

[0031] When the prediction result is consistent with the correct result, a specific negative sample generation process needs to be performed. For the kth argument reference in the current event Select the word with sequence labeling error in the argument reference s k , which represents the position of the word in the text, and then replace the correct label corresponding to the error label to form an error label sequence negative sample The negative sample set is The specific negative sampling process is as follows:

[0032]

[0033]

[0034]

[0035] Where onehot is an operation of converting integer index to one-hot encoding vector, and mid is a median operation.

[0036] The contrast learning regularization step includes:

[0037] For each label sequence L∈{L gold ,L neg ,L pred}, representation learning is performed, where L={l0,l1,…,l n}; for the BIO label, set the label parameter matrix W L to be trained, where each column corresponds to the feature vector of each label in BIO, and according to the matrix W L , the representation Q={q0,q1,…,q n} of each label in each position of the label sequence can be obtained:

[0038] q i =l i ·W L

[0039] After obtaining the label representation Q in the sequence, a linear layer is used to fuse the information of the label and the word vector to obtain the semantic representation of the word with fused label information U={u0,u1,…,u n}.

[0040] u i =W5·(W4·[h i ||q i ]+b4)+b5

[0041] Wherein W4, W5, b4, b5 are linear transformation matrix and corresponding bias term, and || is a vector splicing operation.

[0042] The sequence representation learning of U is carried out using the Transformer, so as to obtain the representation vector Z corresponding to each word and label, Z={z0, z1,..., z n}, and the mean of the output position vector is taken as the final vector representation O of the sequence, O belongs to {O pred , O gold , O neg}:

[0043] Z=Transformer(U)

[0044]

[0045] The triplet margin loss function is used as the loss function for constructing the contrast task

[0046]

[0047] Wherein margin is a hyperparameter, and its meaning means that the distance from the predicted label sequence position to the error sequence position in the expression space should not be less than the difference between the distance from the correct sequence position; the loss functions of the sequence labeling task and the contrast learning regularization task are jointly trained, and wherein alpha, beta are hyperparameters:

[0048]

[0049] The event argument detection step comprises:

[0050] The greedy decoding method is used for label labeling, and the obtained L greedy is taken as the final label sequence, and the word sequence corresponding to the B at the beginning and the continuous I labels immediately after in the label sequence is taken as each argument

[0051] The application further provides an event argument detection system based on label sequence consistency modeling, comprising:

[0052] A training corpus preprocessing module is used to obtain training corpus of annotated event argument roles and event types, perform word segmentation on the text of the training corpus, and obtain the ID of each word in the pretraining dictionary of the language representation model BERT according to the pretraining dictionary.

[0053] a word sequence semantic encoding module, which inputs a word sequence composed of all word IDs into a multi-layer translation model Transformer model of BERT to pre-encode the subword sequence, map the event type into a distributed expression vector, and splice the expression vector with each word vector respectively and fuse them through a linear network to obtain a word semantic expression vector fused with event type information;

[0054] a word label sequence labeling module, which inputs the word semantic expression vector into a full connection network to obtain a probability distribution of each word semantic expression vector belonging to each argument role category, and selects an argument role category with the highest probability in the probability distribution as a predicted argument role category;

[0055] an error label sequence generation module, which divides the word sequence into a correct label sequence predicted correctly and an error label sequence predicted with errors according to the predicted argument role category of each word and the annotated argument role category;

[0056] a contrastive learning regularization module, which performs representation learning on the error label sequence and the correct label sequence, and trains the full connection network and the Transformer model by taking the loss as a regularization term;

[0057] an argument detection module, which inputs a text to be detected and its event type into the trained Transformer model and full connection network in sequence to obtain an argument role category of the text.

[0058] The event argument detection system based on label sequence consistency modeling, wherein the word sequence semantic encoding module comprises:

[0059] a BERT language model pre-trained through a corpus to pre-encode an input word sequence T to obtain more rich dynamic semantic expressions C={c0, c1, …, c n};

[0060] a semantic information of an input event type E is separately encoded, a parameter matrix V is used as an expression vector of each event type to participate in model training, a vector corresponding to the event type is spliced with an output vector of BERT to obtain an intermediate expression X={x0, x1, …, x n}, information of the two is interactively fused through a full connection network to obtain a final context vector representation H={h0, h1, …, h n} of each word fused with event information, and the overall calculation process is as follows:

[0061] x i =[c i ||V(E)]

[0062] h i =W2·(ReLU(W1·x i +b1))+b2

[0063] V(E) represents the vector expression corresponding to the event type E in the parameter matrix V, || is the vector concatenation operation, W1, W2, b1, b2 are linear transformation matrix and corresponding bias term; ReLU is an activation function.

[0064] The event argument detection system based on label sequence consistency modeling, wherein the word label sequence labeling module comprises:

[0065] For each word fusion event information context representation h i , use a linear layer to predict its label probability distribution P = {p0, p1, …, p n}, p i represents the label probability distribution vector of the i-th word:

[0066] z i =W3·h i +b3

[0067]

[0068] Where W3, b3 are linear transformation matrix and corresponding bias term respectively, and a 3-dimensional vector is obtained after transformation, j, k belong to {0, 1, 2} respectively representing the index corresponding to the label O, B, I, represents the probability that the i-th word is labeled as the k-th label; the local predicted label corresponding to the input text Where represents the soft label corresponding to the i-th word, which is a vector of length 3.

[0069] Calculate the loss cross-entropy loss function corresponding to the optimization sequence labeling part As the loss function corresponding to the sequence labeling task:

[0070]

[0071] Where represents the true label corresponding to the i-th word at the j-th position.

[0072] The event argument detection system based on label sequence consistency modeling, wherein the error-prone label sequence generation module comprises:

[0073] The standard correct label sequence is converted to Where is the correct label corresponding to the i-th word;

[0074] When the local prediction label L pred The hard label sequence L is obtained by greedy decoding greedy When the sequence is inconsistent with the correct label sequence, the sequence is generated as an error label negative sample, and the negative sample set L neg greedy According to the local prediction soft label P obtained by the sequence labeling module, the process of greedy decoding includes:

[0075]

[0076]

[0077] When the prediction result is consistent with the correct result, a specific negative sample generation process needs to be performed, and for the kth argument reference in the current event Select the word in the sequence labeling error word in the argument reference s k Indicates the position of the word in the text, and then replaces the correct label corresponding to the error label to form an error label sequence negative sample The negative sample set is The specific negative sampling process is as follows:

[0078]

[0079]

[0080]

[0081] Where onehot is an operation of converting an integer index to a one-hot encoding vector, and mid is a median operation.

[0082] The contrast learning regularization module includes:

[0083] Each label sequence L∈{L gold ,N neg ,L pred} is subjected to representation learning, where L={l0,l1,…,l n}; For the BIO label, set the label parameter matrix W L to be trained, where each column corresponds to the feature vector of each label in BIO, and according to the matrix W L , the representation Q={q0,q1,…,q n} of each position label in each label sequence can be obtained:

[0084] q i =l i ·W L

[0085] ​After obtaining the label expression Q in the sequence, a linear layer is used to fuse the label with the information of the word vector to obtain the word semantic expression U = {u0, u1, …, u n} fused with the label information:

[0086] u i = W5·(W4·[h i ||q i ]+b4)+b5

[0087] Wherein W4, W5, b4, b5 are linear transformation matrix and corresponding bias term, and || is a vector splicing operation.

[0088] The sequence representation learning of U is performed using the Transformer to obtain the expression vector corresponding to each word and label Z = {z0, z1, …, z n}, and the mean of the output position vector is taken as the final vector representation of the sequence O ∈ {O pred ,O gold ,O neg}:

[0089] Z = Transformer (U)

[0090]

[0091] The triplet margin loss function is used as the loss function for constructing the contrast task

[0092]

[0093] Wherein margin is a hyperparameter, and its meaning means that the distance from the predicted label sequence position to the error sequence position and the distance from the correct sequence position should not be less than margin; the loss functions of the sequence labeling task and the contrast learning regularization task are jointly trained, and wherein alpha, beta are hyperparameters:

[0094]

[0095] The event argument detection module comprises:

[0096] The greedy decoding method is used for label annotation, and the obtained L greedy is taken as the final label sequence, and the word sequence corresponding to the B at the beginning of the label sequence and the subsequent continuous I labels is taken as the argument

[0097] The application further provides a storage medium for storing a program for executing any one of the event argument detection methods based on label sequence consistency modeling.

[0098] The application further provides a client for any one of the event argument detection systems based on label sequence consistency modeling.

[0099] From the above scheme, the application has the advantages that:

[0100] First, contrast learning is used to improve the internal consistency of word label sequences, and to alleviate the few recalls and incorrect detection problems caused by local label errors and omissions; second, the contrast learning task loss is used as a regularization term for joint training, to alleviate the overfitting problem of single sequence labeling; third, the performance on the event argument detection task is improved, and the event argument detection F1 value of the application on the RAMS public data test set reaches 44.2%, which is better than the existing BERT-based sequence labeling technology, and the F1 value of the technology is only 39.3%. BRIEF DESCRIPTION OF DRAWINGS

[0101] Figure 1 The overall flowchart of the application is shown in the figure;

[0102] Figure 2 The overall method structure diagram in the application is shown in the figure. DETAILED DESCRIPTION

[0103] In order to overcome the deficiencies in the prior art, the application provides an event argument detection method based on label sequence consistency modeling. A specific sampling strategy is used to generate error label sequences that meet typical error characteristics according to correct label sequences, and then representation learning is performed on the correct and error label sequences respectively. Through the method of contrast learning, the model is modeled for the consistency of the labels from the contrast learning of positive and negative samples. In addition, by jointly learning the contrast learning task and the original sequence labeling task, the overfitting problem caused by the single sequence labeling model is alleviated, thereby improving the effect of argument detection.

[0104] The event argument role prediction method provided by the application includes the following steps:

[0105] 1) Training corpus preprocessing: the training corpus used by the application is selected from the RAMS data set, and the processing process is to use the Word Piece method to perform word segmentation on the text, then convert each word into the ID corresponding to the BERT pre-training dictionary, add the self-defined [Event] special label before and after the trigger word corresponding position, and finally add the [CLS] and [SEP] special labels consistent with BERT and the training task at the beginning and end of the sentence respectively;

[0106] 2) Word sequence semantic encoding: using BERT pre-training language model for pre-encoding, inputting the word id sequence processed in the last step into the multi-layer Transformer model of BERT to pre-encode the sub-word sequence, using the language model BERT pre-trained by large-scale corpus to obtain the semantic features of the word, compared with the traditional static word and word vector, the BERT trained by large-scale corpus can provide more dynamic semantic expression. Then, the correct event type is mapped to a learnable distributed representation vector, which is spliced with each word vector respectively and fused by a linear network to obtain a word semantic expression vector fused with event type information.

[0107] 3) Word label sequence labeling: using BIO label system, using a fully connected network to predict the probability distribution of each word belonging to each label in the BIO label system.

[0108] 4) Error-prone label sequence generation: according to the prediction probability distribution, sampling the error-prone label sequence. Among them, the prediction does not conform to the preset label category, that is, the prediction fails, which belongs to the confusion.

[0109] 5) Contrastive learning regularization: using the aforementioned transformer model to encode the semantic of the label sequence to obtain the semantic representation vector, learning the representation of the error label sequence and the correct label sequence, constructing a contrastive learning task as a regularization term to improve the consistency of the word sequence label.

[0110] In order to make the above features and effects of the present application more clear and easy to understand, the following embodiments are described in detail below, and the accompanying drawings are described as follows.

[0111] The present application proposes an event argument detection method based on label sequence consistency modeling, and the overall process of the method is as shown in Figure 1 The method mainly includes word sequence semantic encoding, word label sequence labeling, error-prone label sequence generation, and contrastive learning regularization. The word sequence semantic encoding uses BERT and training language model to learn the semantic representation of the preprocessed word, and integrates the event type information into the representation vector; the word label sequence labeling uses a fully connected network to estimate the label probability distribution corresponding to each word; the error-prone label sequence generation generates an error-prone label sequence according to a certain strategy according to the word label sequence probability distribution; the contrastive learning regularization is based on the contrastive learning of the error-prone label sequence and the correct label sequence to construct a regularization loss, and improve the consistency of the word sequence label. The specific method steps include:

[0112] S1, pre-encode using a BERT pre-trained language model, add self-defined [Event] special tags before and after the trigger word corresponding position, and add [CLS] and [SEP] special tags at the beginning and end of the sentence respectively, which are consistent with the training task of BERT, mark the beginning and sentence position, and use it here to keep consistent with it, so as to obtain more accurate semantic features.

[0113] The processed word sequence is input into the multi-layer Transformer model of BERT to pre-encode the word sequence. Then, the event type is mapped to a learnable distributed representation vector through lookup embedding, and is spliced with each word vector respectively and fused through a linear network to obtain a word semantic expression vector fused with event type information. The word vector refers to the embedding corresponding to each word position obtained by BERT language model coding.

[0114] S2, based on the BIO label system, use a fully connected network to predict the probability distribution of each word semantic expression vector belonging to each label, use ReLU as the activation function, and use the Softmax function to model the probability distribution.

[0115] S3, according to the estimated probability distribution, sample to generate an easily confused error label sequence.

[0116] S4, perform representation learning on the error label sequence and the correct label sequence, construct a contrastive learning task, and use the loss as a regularization term to participate in training to improve the consistency of the word sequence label, and use the triplet margin loss here.

[0117] Specifically, S1 includes 3 sub-steps as follows.

[0118] S101, training data preprocessing. The text is segmented by the WordPieceTokenizer module in the Transformers library, self-defined [Event] special tags are added before and after the trigger word corresponding position, and then [CLS] and [SEP] special tags consistent with the training task of BERT are added at the beginning and end of the sentence respectively, and the input of the same batch of words is padded to the same length according to the longest text length of the batch of words.

[0119] S102, BERT pre-training model coding. The BERT language model pre-trained by large-scale corpus is used to pre-code the input word sequence T, compared with the traditional static word vector, this method can obtain more rich dynamic semantic expression C={c0,c1,…,c n},c n is the dynamic semantic expression of the nth word.

[0120] C = BERT(T)

[0121] S103. Semantic information of input event type E is encoded separately, and a separate trainable parameter matrix V is used as the representation vector of each event type to participate in model training. Then, the vectors corresponding to the event types are concatenated with the output vector of BERT to obtain the intermediate representation X = {x0, x1, ..., x...} n}, x n The intermediate semantic representation of the nth word is used, and finally, the information from both is fused through a fully connected network to obtain the context vector representation H = {h0, h1, ..., hn} of the fused event information for each word. n The overall calculation process is as follows:

[0122] x i =[c i ||V(E)]

[0123] h i =W2·(ReLU(W1·x) i +b1))+b2

[0124] V(E) represents the vector representation of event type E in the parameter matrix V. || is the vector concatenation operation. W1, W2, b1, and b2 are the linear transformation matrices and their corresponding bias terms. ReLU is a nonlinear rectifier, used here as the activation function.

[0125] S2 also includes 3 sub-steps as follows.

[0126] S201, For each word, the context representation h of the fused event information i This invention also uses a linear layer to predict its label probability distribution P = {p0, p1, ..., p...} n}, p i Let represent the probability distribution vector of the label for the i-th word, calculated as follows:

[0127] z i =W3·h i +b3

[0128]

[0129] Where W3 and b3 are the linear transformation matrix and the corresponding bias term, respectively, the transformation yields a 3D vector, and j, k belong to {0,1,2}, representing the indices corresponding to labels O, B, and I, respectively. This represents the probability that the i-th word is labeled as the k-th label. After calculation, the locally predicted label corresponding to the input text can be obtained. The predicted labels here are designed to be soft labels, where denotes the soft label corresponding to the i-th word, which is a vector of length 3.

[0130] S202, calculate the loss cross-entropy loss function corresponding to the optimization sequence labeling part as the loss function corresponding to the sequence labeling task.

[0131]

[0132] wherein denotes the true label corresponding to the i-th word in the j-th position.

[0133] Similarly, S3 is also divided into two steps in particular.

[0134] S301, correct label sequence processing. The standard correct label sequence is converted to wherein is the correct label corresponding to the i-th word, and the local predicted label L pred The difference is that the hard label representation method is used here, that is, the one-hot encoding vector, such as the label vector corresponding to the B label is [0, 1, 0], and the vector corresponding to the I label is [0, 0, 1].

[0135] S302, error label sequence generation. When the predicted soft label L pred The hard label sequence L greedy is obtained by greedy decoding. When the correct label sequence is inconsistent, the sequence is directly generated as the error label negative sample, that is, no additional negative sampling is needed, and the negative sample set L neg = L greedy According to the local predicted soft label P obtained by the sequence labeling module, the process of greedy decoding is as follows:

[0136]

[0137]

[0138] When the prediction result is consistent with the correct result, a specific negative sample generation process is needed, that is, for the k-th argument referent in the current event b is the beginning position (begin) of the argument referent, e is the end position (end) of the argument referent, a indicates that the position belongs to the argument (argument), rather than the trigger word, and A k The word in the argument referent that is most likely to be mislabeled by the sequence labeling S k denotes the position of the word in the text, and then the correct label corresponding to the word is replaced by the most confusing error label, thereby forming an error label sequence negative sample The negative sample set is The specific negative sampling process is as follows:

[0139]

[0140]

[0141]

[0142] Wherein, onehot is an operation of converting an integer index into a one-hot encoding vector, and mid is a median operation. When the probabilities of the two labels corresponding to the most probable word are closer, it means that the judgment of the label at this position is less confident, which is easy to cause prediction errors, so the word corresponding to the least confident label is selected and the label is replaced with another label that is easy to confuse the model.

[0143] Finally, for the S4 step, for the specific construction process, first, the present application performs representation learning on each label sequence L∈{L gold ,L neg ,L pred} where L={l0,l1,…,l n}. For the BIO label, the present application sets a trainable label parameter matrix W L , wherein each column corresponds to the feature vector of each label in BIO. According to the matrix, the representation Q={q0,q1,…,q n} of each label in the label sequence can be obtained:

[0144] q i =l i ·W L

[0145] After obtaining the label representation Q in the sequence, a linear layer is used to fuse the information of the label and the word vector to obtain the semantic representation of the word U={u0,u1,…,u n} with fused label information:

[0146] u i =W5·(W4·[h i ||q i ]+b4)+b5

[0147] Wherein W4, W5, b4, b5 are linear transformation matrices and corresponding bias terms, and || is a vector concatenation operation.

[0148] Finally, the present application uses the Transformer to perform sequence representation learning on U to obtain the representation vector Z={z0,z1,…,z n} as the final vector representation of the sequence O e {O pred ,O gold ,O neg}:

[0149] Z = Transformer(U)

[0150]

[0151] In order to make O pred close to O gold and far from O neg , the application uses a triplet margin loss function as the loss function for constructing the contrast task:

[0152]

[0153] Where margin is a hyperparameter, and its meaning means that the distance from the predicted label sequence position to the wrong sequence position in the expression space should not be less than the difference between the distance from the correct sequence position. Finally, the application uses the loss function of both the sequence labeling task and the contrast learning regularization task for joint training, where alpha, beta are hyperparameters:

[0154]

[0155] In the prediction stage, the application uses a greedy decoding method for label labeling, and the obtained L greedy is taken as the final label sequence, and the word sequence corresponding to the B at the beginning of the label sequence and the continuous I labels immediately after it is taken as the argument Figure 2 decoded by the application.

[0156] The following is a system embodiment corresponding to the above method embodiment, and the present embodiment can be implemented in cooperation with the above embodiments. The related technical details mentioned in the above embodiments are still valid in the present embodiment. In order to reduce repetition, they will not be repeated here. Correspondingly, the related technical details mentioned in the present embodiment can also be applied in the above embodiments.

[0157] The application also proposes an event argument detection system based on label sequence consistency modeling, which comprises:

[0158] A training corpus preprocessing module is used to obtain training corpus of annotated event argument roles and event types, perform word segmentation on the text of the training corpus, and obtain the ID of each word in the pretraining dictionary according to the pretraining dictionary of the language representation model BERT;

[0159] The word sequence semantic encoding module inputs the word sequence composed of all word IDs into a multi-layer translation model Transformer model of BERT to pre-encode the sub-word sequence, map the event type into a distributed expression vector, and splice the expression vector with each word vector respectively and fuse them through a linear network to obtain a word semantic expression vector fused with event type information;

[0160] The word label sequence labeling module inputs the word semantic expression vector into a full connection network to obtain a probability distribution of each word semantic expression vector belonging to each event argument role category, and selects an event argument role category with the highest probability in the probability distribution as a predicted argument role category;

[0161] The error label sequence generation module divides the word sequence into a correct label sequence predicted correctly and an error label sequence predicted with errors according to the predicted argument role category of each word and the annotated event argument role category;

[0162] The contrast learning regularization module performs representation learning on the error label sequence and the correct label sequence, and takes the loss as a regularization term to train the full connection network and the Transformer model;

[0163] The event argument detection module inputs the text of the event argument to be detected and its event type into the trained Transformer model and full connection network in sequence to obtain the argument role category of the text.

[0164] The event argument detection system based on label sequence consistency modeling, wherein the word sequence semantic encoding module comprises:

[0165] The BERT language model pre-trained through a corpus is used to pre-encode the input word sequence T to obtain more rich dynamic semantic expression C={c0, c1, …, c n};

[0166] The semantic information of the input event type E is separately encoded, a parameter matrix V is used as an expression vector of each event type to participate in model training, the vector corresponding to the event type is spliced with the output vector of BERT to obtain an intermediate expression X={x0, x1, …, x n}, the information of the two is interactively fused through a full connection network to obtain a final context vector representation H={h0, h1, …, h n} of each word fused with event information, and the overall calculation process is as follows:

[0167] x i =[c i ||V(E)]

[0168] h i= W2 • (ReLU(W1 • x i + b1) ) + b2

[0169] V(E) represents the vector expression corresponding to the event type E in the parameter matrix V, || is the vector concatenation operation, W1, W2, b1, b2 are linear transformation matrix and corresponding bias term; ReLU is an activation function.

[0170] The event argument detection system based on label sequence consistency modeling, wherein the word label sequence labeling module comprises:

[0171] For each word fusion event information context representation h i , use a linear layer to predict its label probability distribution P = {p0, p1, …, p n}, p i represents the label probability distribution vector of the i-th word:

[0172] z i = W3 • h i + b3

[0173]

[0174] Where W3, b3 are linear transformation matrix and corresponding bias term respectively, and a 3-dimensional vector is obtained after transformation, j, k belong to {0, 1, 2} respectively representing the index corresponding to the label O, B, I, represents the probability that the i-th word is labeled as the k-th label; the local predicted label Where represents the soft label corresponding to the i-th word, which is a vector of length 3.

[0175] Calculate the loss cross-entropy loss function corresponding to the optimization sequence labeling part As the loss function corresponding to the sequence labeling task:

[0176]

[0177] Where represents the true label corresponding to the i-th word at the j-th position.

[0178] The event argument detection system based on label sequence consistency modeling, wherein the error-prone label sequence generation module comprises:

[0179] The standard correct label sequence is converted to Where is the correct label corresponding to the i-th word;

[0180] When the local predicted label L predThe hard label sequence L is obtained by greedy decoding greedy When the sequence is inconsistent with the correct label sequence, the sequence is taken as a generated error label negative sample, and a negative sample set L neg greedy is obtained according to the local prediction soft label P obtained by the sequence labeling module, and the process of greedy decoding includes:

[0181]

[0182]

[0183] When the prediction result is consistent with the correct result, a specific negative sample generation process needs to be performed, for the kth argument reference in the current event Select the word in the sequence labeling error word in the argument reference s k Indicates the position of the word in the text, and then replaces the correct label corresponding to the word with an error label, thereby forming an error label sequence negative sample The negative sample set is The specific negative sampling process is as follows:

[0184]

[0185]

[0186]

[0187] Where onehot is an operation of converting an integer index into a one-hot encoding vector, and mid is a median operation.

[0188] The contrast learning regularization module includes:

[0189] Each label sequence L∈{L gold ,L neg ,L pred} is subjected to representation learning, where L={l0,l1,…,l n}; for the BIO label, a label parameter matrix W L to be trained is set, where each column corresponds to a feature vector of each label in BIO, and according to the matrix W L , the representation Q={q0,q1,…,q n} of each label in each position of the label sequence can be obtained:

[0190] q i =l i ·W L

[0191] ​After obtaining the label expression Q in the sequence, a linear layer is used to fuse the label with the information of the word vector to obtain the word semantic expression U = {u0, u1, …, u n} fused with the label information:

[0192] u i = W5·(W4·[h i ||q i ]+b4)+b5

[0193] Wherein W4, W5, b4, b5 are linear transformation matrix and corresponding bias term, || is a vector splicing operation.

[0194] The sequence representation learning of U is performed using the Transformer to obtain the expression vector corresponding to each word and label Z = {z0, z1, …, z n}, and the mean of the output position vector is taken as the final vector representation of the sequence O ∈ {O pred ,O gold ,O neg}:

[0195] Z = Transformer (U)

[0196]

[0197] The triplet margin loss function is used as the loss function for constructing the contrast task

[0198]

[0199] Wherein margin is a hyperparameter, and its meaning means that the distance from the predicted label sequence position to the error sequence position and the distance from the correct sequence position should not be less than margin; the loss functions of the sequence labeling task and the contrast learning regularization task are jointly trained, and wherein alpha, beta are hyperparameters:

[0200]

[0201] The event argument detection module comprises:

[0202] The greedy decoding method is used for label annotation, and the obtained L greedy is taken as the final label sequence, and the word sequence corresponding to the B at the beginning of the label sequence and the continuous I labels immediately after is taken as the argument

[0203] The application further provides a storage medium for storing a program for executing any one of the event argument detection methods based on label sequence consistency modeling.

[0204] The present application also provides a client for any one of the event-based ontology detection systems based on the consistency modeling of label sequences.

Claims

1. An event argument detection method based on label sequence consistency modeling, characterized in that, include: The training corpus preprocessing steps involve obtaining training corpus labeled with event argument roles and event types, segmenting the text in the training corpus into words, and obtaining the ID of each word in the pre-trained dictionary based on the language representation model BERT. The word sequence semantic encoding step involves inputting the word sequence composed of all word IDs into the BERT multilayer translation model Transformer to pre-encode the sub-word sequence, mapping the event type to a distributed expression vector, concatenating it with each word vector, and fusing it through a linear network to obtain a word semantic expression vector that integrates event type information. The word tag sequence annotation step involves inputting the semantic representation vector of a word into a fully connected network to obtain the probability distribution of the semantic representation vector of each word belonging to each event argument role category. The event argument role category with the highest probability in this probability distribution is selected as the predicted argument role category. The error-prone label sequence generation step divides the word sequence into a correct label sequence and an incorrect label sequence based on the predicted argument role category and the labeled event argument role category for each word. By comparing the learning regularization steps, representation learning is performed on the incorrect label sequence and the correct label sequence, and the loss is used as a regularization term to train the fully connected network and the Transformer model; The event argument detection step involves sequentially inputting the text of the event arguments to be detected and their event types into the trained Transformer model and the fully connected network to obtain the argument role category of the text. The semantic encoding steps for this word sequence include: By pre-training the BERT language model on the corpus, the input character and word sequence T is pre-encoded to obtain richer dynamic semantic expressions. ; The semantic information of the input event type E is encoded separately, and the parameter matrix V is used as the representation vector of each event type to participate in model training. The vector corresponding to the event type is concatenated with the output vector of BERT to obtain the intermediate representation. By using a fully connected network to interact and fuse the information from both, a context vector representation of the fused event information for each word is obtained. The overall calculation process is as follows: V(E) represents the vector representation of event type E in parameter matrix V; || is the vector concatenation operation; W1, W2, b1, b2 are linear transformation matrices and their corresponding bias terms; ReLU is the activation function. The word tag sequence annotation steps include: Contextual representation of event information for each word Predicting the label probability distribution using a linear layer , The vector representing the probability distribution of the label for the i-th word: in These are the linear transformation matrix and the corresponding bias term, respectively. The transformation yields a 3D vector. The values ​​in {0, 1, 2} represent the indices corresponding to labels O, B, and I, respectively. This represents the probability that the i-th word is labeled as the k-th label; the local predicted label corresponding to the input text. },in The soft tag corresponding to the i-th word is a vector of length 3; Calculate the loss, cross-entropy, and loss function corresponding to the optimized sequence labeling part. As the loss function for sequence labeling tasks: in This represents the actual label corresponding to the i-th word at the j-th position; The steps for generating this error-prone tag sequence include: Standard correct label sequence converted to ;in The correct label for the i-th word; When local prediction label The hard tag sequence is obtained through greedy decoding. When the sequence is inconsistent with the correct label sequence, the sequence is used as a negative sample with an incorrect label. The negative sample set... Based on the locally predicted soft label P obtained from the sequence labeling module, the greedy decoding process includes: When the prediction result matches the correct result, a specific negative sample generation process is required, for the k-th argument in the current event. Select words with incorrect sequence labeling within their argument references. , The position of the word in the text is indicated, and then its corresponding correct label is replaced with the incorrect label, thus forming a negative sample of the incorrect label sequence. The negative sample set is The specific negative sampling process is as follows: Where onehot is the operation of converting integer indices into one-hot encoded vectors, and mid is the operation of taking the median; The contrastive learning regularization steps include: For each tag sequence Representation learning is performed, where For BIO tags, set the tag parameter matrix to be trained. Each column corresponds to a feature vector of each tag in BIO, based on this matrix. It can obtain the representation of the label at each position in each label sequence. : After obtaining the label representation Q from the sequence, a linear layer is used to fuse the label and word vector information to obtain the word semantic representation with fused label information. : in , , , Let be the linear transformation matrix and its corresponding bias term, and || be the vector concatenation operation; Using Transformer, sequence representation learning is performed on U to obtain the representation vectors corresponding to each word and label. The mean of the output vectors at each position is used as the final vector representation of the sequence. : The ternary margin loss function is used as the loss function for constructing the contrast task. : Here, `margin` is a hyperparameter, meaning that the difference between the distance from the predicted label sequence position to the incorrect sequence position and its distance to the correct sequence position in the representation space should not be less than `margin`. The loss functions of both the sequence labeling task and the contrastive learning regularization task are used for joint training. For hyperparameters: The event argument detection steps include: Use a greedy decoding method to label the tags, and get As the final label sequence, the word sequence corresponding to the first B followed by several consecutive I labels in the label sequence is used as the arguments obtained from decoding. .

2. An event argument detection system based on label sequence consistency modeling, characterized in that, include: The training corpus preprocessing module obtains the training corpus with labeled event argument roles and event types, segments the text in the training corpus into words, and obtains the ID of each word in the pre-trained dictionary based on the language representation model BERT. The word sequence semantic encoding module inputs the word sequence composed of all word IDs into the BERT multilayer translation model Transformer to pre-encode the sub-word sequence, maps the event type into a distributed expression vector, and concatenates it with each word vector and fuses them through a linear network to obtain a word semantic expression vector that integrates event type information. The word tag sequence annotation module inputs the semantic representation vector of a word into a fully connected network to obtain the probability distribution of the semantic representation vector of each word belonging to each event argument role category. The event argument role category with the highest probability in the probability distribution is selected as the predicted argument role category. The error-prone label sequence generation module divides the word sequence into a correct label sequence and an incorrect label sequence based on the predicted argument role category and the labeled event argument role category for each word. The contrastive learning regularization module performs representation learning on the incorrect label sequence and the correct label sequence, and uses its loss as a regularization term to train the fully connected network and the Transformer model; The event argument detection module takes the text of the event arguments to be detected and their event types and inputs them sequentially into the trained Transformer model and the fully connected network to obtain the argument role category of the text. The word sequence semantic encoding module includes: By pre-training the BERT language model on the corpus, the input character and word sequence T is pre-encoded to obtain richer dynamic semantic expressions. ; The semantic information of the input event type E is encoded separately, and the parameter matrix V is used as the representation vector of each event type to participate in model training. The vector corresponding to the event type is concatenated with the output vector of BERT to obtain the intermediate representation. By using a fully connected network to interact and fuse the information from both, a context vector representation of the fused event information for each word is obtained. The overall calculation process is as follows: V(E) represents the vector representation of event type E in parameter matrix V; || is the vector concatenation operation; W1, W2, b1, b2 are linear transformation matrices and their corresponding bias terms; ReLU is the activation function. The word tag sequence annotation module includes: Contextual representation of event information for each word Predicting the label probability distribution using a linear layer , The vector representing the probability distribution of the label for the i-th word: in These are the linear transformation matrix and the corresponding bias term, respectively. The transformation yields a 3D vector. The values ​​in {0, 1, 2} represent the indices corresponding to labels O, B, and I, respectively. This represents the probability that the i-th word is labeled as the k-th label; the local predicted label corresponding to the input text. },in The soft tag corresponding to the i-th word is a vector of length 3; Calculate the loss, cross-entropy, and loss function corresponding to the optimized sequence labeling part. As the loss function for sequence labeling tasks: in This represents the actual label corresponding to the i-th word at the j-th position; The error-prone label sequence generation module includes: Standard correct label sequence converted to ;in The correct label for the i-th word; When local prediction label The hard tag sequence is obtained through greedy decoding. When the sequence is inconsistent with the correct label sequence, the sequence is used as a negative sample with an incorrect label. The negative sample set... Based on the locally predicted soft label P obtained from the sequence labeling module, the greedy decoding process includes: When the prediction result matches the correct result, a specific negative sample generation process is required, for the k-th argument in the current event. Select words with incorrect sequence labeling within their argument references. , The position of the word in the text is indicated, and then its corresponding correct label is replaced with the incorrect label, thus forming a negative sample of the incorrect label sequence. The negative sample set is The specific negative sampling process is as follows: Where onehot is the operation of converting integer indices into one-hot encoded vectors, and mid is the operation of taking the median; This contrastive learning regularization module includes: For each tag sequence Representation learning is performed, where For BIO tags, set the tag parameter matrix to be trained. Each column corresponds to a feature vector of each tag in BIO, based on this matrix. It can obtain the representation of the label at each position in each label sequence. : After obtaining the label representation Q from the sequence, a linear layer is used to fuse the label and word vector information to obtain the word semantic representation with fused label information. : in , , , Let be the linear transformation matrix and its corresponding bias term, and || be the vector concatenation operation; Using Transformer, sequence representation learning is performed on U to obtain the representation vectors corresponding to each word and label. The mean of the output vectors at each position is used as the final vector representation of the sequence. : The ternary margin loss function is used as the loss function for constructing the contrast task. : Here, `margin` is a hyperparameter, meaning that the difference between the distance from the predicted label sequence position to the incorrect sequence position and its distance to the correct sequence position in the representation space should not be less than `margin`. The loss functions of both the sequence labeling task and the contrastive learning regularization task are used for joint training. For hyperparameters: The event argument detection module includes: Use a greedy decoding method to label the tags, and get As the final label sequence, the word sequence corresponding to the first B followed by several consecutive I labels in the label sequence is used as the arguments obtained from decoding. .

3. A storage medium for storing a program that executes the event argument detection method based on tag sequence consistency modeling as described in claim 1.

4. A client for the event argument detection system based on label sequence consistency modeling as described in claim 2.

Images