An emergency argument extraction method for multi-modal data

Abstract

The application discloses a new event argument extraction method based on machine reading comprehension technology, which is mainly used for extracting event arguments under the condition that the type of a sudden event is known. The method mainly includes two parts. One part is an image description generation module. The module uses ResNet to extract image features for a microblog image containing a sudden event, and then adopts a beam search method to decode a text description about the microblog image from the image features. Through the method, the information of the image is introduced, and the input of the model is enhanced. The other part is an argument extraction module. The module generates a question by using the identified type of the sudden event and the event argument to be extracted, and extracts the required event argument from the microblog text and the image description by using a machine reading comprehension method.

Description

TECHNICAL FIELD

[0001] The application relates to a sudden event argument extraction method based on a machine reading comprehension technology and for multi-modal data, and belongs to the technical field of natural language processing. BACKGROUND

[0002] In today's information society, the Internet has become an important part of people's life. According to the China Internet Development Status Statistical Report, as of June 2021, the number of Chinese netizens reached 1.011 billion, and the Internet penetration rate reached 71.6%. Among them, the proportion of Chinese netizens using mobile phones to access the Internet reached 99.6%. The rapid development of the Internet has led to a rapid increase in the generation, transmission and acquisition of information, and the proportion of text and video is also increasing. From the vast amount of multi-modal information, it is very important to quickly and accurately locate and extract truly valuable information. Information extraction technology deeply affects the speed of people obtaining information.

[0003] Event extraction is a key technology of information extraction, which is used to detect events from a corpus and extract key information of the events, such as time, place, person, etc. For example, "this morning, a US Navy drone took off from Andersen Air Force Base in Guam to carry out a reconnaissance mission" contains a plane reconnaissance task event, and the event arguments are: time "this morning", place "Andersen Air Force Base in Guam", and event subject "US Navy drone".

[0004] In recent years, the development of event extraction technology has benefited from the accumulation of high-quality data and the development of mainstream models. Social platforms, as a new medium, provide a vast source of data for event extraction technology and pose new challenges. A significant feature of social networks is that everyone can report on sudden events. Sudden events generally include natural disasters, accidents, public health events and social security events, such as fires, floods, epidemics, etc., which interfere with people's stable life, so it is urgent to accurately identify sudden events in a timely manner. With the development of the network, the information created by users also grows exponentially, which easily leads to interaction between netizens. Sudden events will attract netizens to discuss and exchange, constantly ferment and spread rapidly, easily causing network public opinion and possibly influencing the development of the event and triggering another social sudden event.

[0005] In the traditional event extraction method, discrete features constructed manually are often used to represent candidate trigger words and candidate trigger-argument pairs. According to the length of the input, the traditional event extraction method can be divided into sentence-level event extraction method and document-level event extraction method. The sentence-level event extraction method considers the word-based features and the word pair-based features. The word-based discrete features include: (1) morphological features, including unigram and bigram grammar, i.e. considering the current word and adjacent words, synonyms, word forms, stems, etc.; (2) syntactic features, including the dependent words and core words of the current word, the syntactic tree path involved, the dependency relationship, etc.; (3) entity features, including entity types, the closest entity type, etc. The word pair-based discrete features include: various dependency relationships between trigger words and arguments, and co-occurrence relationships between trigger words and trigger words, etc. The document-level event extraction method additionally considers cross-sentence or even cross-document features. For example, event consistency features of entities of the same type participating in the event, event type consistency features of the same word in the document, argument consistency features of similar events, etc. In the machine learning task, the selection of features directly affects the experimental results. Due to the difficulty in obtaining annotated corpus, manual annotation of corpus consumes a lot of effort, which is a defect of this method.

[0006] There are few multi-modal event extraction works, and there is no definite conclusion on how to correctly introduce image features. A few related works have strong assumptions, such as that event arguments need to be one-to-one corresponding in pictures and texts to realize the alignment of image-text semantic space through training, and the expression of sudden events in microblogs often does not conform to the assumption. SUMMARY

[0007] The present application is based on the fact that existing event extraction models are difficult to process image information on social platforms. An image description generation method is used to introduce image information, and the microblog text information is supplemented by describing the picture content, which is modeled as a whole text sequence labeling task, which is more conducive to transferring the background knowledge learned on a large-scale text corpus by using a pre-trained model, and effectively alleviates the problem that the model is difficult to train due to the lack of annotation resources.

[0008] The scheme discloses a kind of emergency argument extraction method for multi-modal data, mainly used in the case where known emergency type, extract event argument.This scheme is mainly divided into two parts, one part is image description generation module, this module uses ResNet to extract image features to the microblog picture containing emergency, then a beam search is used to decode the text description about microblog picture from image features, the information of image is introduced by this method, and the input of model is enhanced.The other part is argument extraction module, the module generates question to the emergency type identified and the event argument to be extracted, and extracts the required event argument from microblog text and image description using machine reading comprehension.

[0009] In order to achieve the above purpose, the present application proposes a kind of emergency argument extraction method for multi-modal data, specific steps are as follows:

[0010] Step 1) extract the image features of microblog picture by using ResNet model;

[0011] Step 2) the image features of microblog picture obtained in step 1) are decoded by LSTM model, and the image description of microblog picture is obtained;

[0012] Step 3) the model in step 2) is trained in a large number of annotated picture-text pair data using Teacher-Forcing, and the text description of image is obtained using beam search when actually predicting;

[0013] Step 4) the event type, argument to be extracted, microblog text information and microblog image description are spliced into natural language chapter and question;

[0014] Step 5) the chapter and question obtained in step 4) are converted into corresponding vector representation by pre-training model BERT;

[0015] Step 6) the required event argument is extracted from the vector representation obtained in step 5) by machine reading comprehension model;

[0016] Step 7) the initial model is trained on a large number of annotated event argument pair data by mini-batch gradient descent, and the optimized event argument extraction model is obtained.

[0017] In step 2), the microblog image representation generated by ResNet is decoded into the description of the image by the decoder, and the problem is modeled as a unidirectional recursive autoregressive model, so that the probability of the generated sequence on the language model is maximum, and the specific implementation is as follows:

[0018] p(Y | X) = p(Y1 | X)p(Y2 | X, Y1)p(Y3 | X, Y1, Y2)p(Y4 | X, Y1, Y2, Y3);

[0019] where X is the input sequence set of the model, X i is the i-th input word, Y is the output sequence set of the model, Y i is the i-th output word

[0020] In step 3), the model can predict the input and output of each step of the decoder in advance in the training stage due to the labeled data pair, so the target sequence can be shifted by one bit for training, which is more convenient for training, and the model is easier to learn the ability to generate corresponding sequences. This training method is called Teacher-Forcing. But this training method is only implemented in training, and does not exist in the prediction stage, so the decoder cannot use the input of the next time step in advance when performing each step.

[0021] In the actual prediction stage, the sequence with the maximum probability can be obtained by enumerating all possible output sequences, but the computational overhead is too large, and this is not a Markov process and cannot be solved using dynamic programming. If the output with the maximum probability is selected each time, it is called greedy search, which has much less computational overhead than enumerating all sequences in the past, but the result obtained by this method is often not optimal. Assuming that the first step selects a word with a probability that is not the maximum, when the second step is entered, a very high conditional probability p(Y2 | X, Y1) may be obtained, so the result of multiplying the two will exceed the greedy algorithm that takes the maximum bit by bit. In order to efficiently and accurately solve this problem, a beam search strategy is used. For each generation process, first, set the hyperparameter k as the beam width (beam size) of beam search. Second, in the first time step, select the k words with the maximum conditional probability in the language model at the current time step to form the first word of the k candidate output sequences. Third, in each subsequent time step, based on the k candidate output sequences of the previous time step, concatenate each word in the candidate dictionary to select the k sequences with the maximum conditional probability from the new possible output sequences as the candidate output sequences of this time step. Finally, filter the candidate output sequences from each time step to select the candidate output sequence containing the special symbol "EOS" as the final output sequence. <eos>" and all special symbols <eos>The subsequences after "" are discarded, resulting in a set of final candidate output sequences.

[0022] In step 4), the event type, arguments to be extracted, Weibo text information, and Weibo image description are concatenated into a natural language passage and question. This step transforms the extraction task into a machine reading comprehension task, and the proper construction of the question plays a crucial role in improving the model's extraction performance. The known event type and arguments to be extracted are packaged into a question using a template, and a special symbol [CLS] is added before the question; the Weibo text is concatenated with the question, separated by a special symbol [SEP]; the image description of the Weibo image is concatenated with the question and text sequence, separated by special tokens [IMG]; the final concatenated sequence is:

[0023] query=[CLS]+query_pattern(event_type,argument)+

[0024] [SEP]+sentence+[IMG]+image_caption;

[0025] [CLS], [SEP], and [IMG] are special markers; query_pattern is the template for generating the question; event_type is the event type; argument is the name of the event argument to be extracted; sentence is the text sequence of the Weibo post; and imagecaption is the text description generated for the Weibo post's accompanying image.

[0026] In step 5), the passages and questions obtained in step 4) are transformed into corresponding vector representations using the pre-trained model BERT, and the transformation is carried out according to the following steps.

[0027] Step 51) Generate Token Embedding, also known as word Embedding. For Chinese, it is segmented by character, and for English, it is preprocessed using WordPiece to split words into affixes, reducing the impact of unregistered words.

[0028] Step 52) Generate Segment Embedding to distinguish the various parts that are spliced ​​together. Mark the question region as 0, the text token region as 1, and the image token region as 2. This can ensure that the final extracted argument span exists in the text sequence.

[0029] Step 53) Generate Position Embedding, encode the position of each input, add position information, and use BERT's bit-by-bit encoding method;

[0030] Step 54) Add the three embedding layers together to obtain the embedding layer;

[0031] Step 55) Input the text embedding layer vector into the standard BERT pre-trained model to obtain the feature representation of the sequence.

[0032] In step 6), the machine reading comprehension model extracts the required event arguments from the vector representation obtained in step 5). Specifically, it extracts the vector at the [CLS] position, uses a fully connected layer to transform the BERT output into a 2-dimensional hidden layer, representing the start and end positions of the argument. The cross-entropy loss function is used to calculate the loss values ​​between the argument and the true label, and the sum and average are taken to obtain the final loss value. Finally, by statistically analyzing the span between the start and end positions, the argument to be extracted can be obtained.

[0033] Compared with the prior art, the advantages of the present invention are as follows:

[0034] 1) For the task of event argument extraction, existing multimodal event extraction methods rely on strong assumptions, such as the need for event arguments to correspond one-to-one in images and text in order to achieve alignment of the image-text semantic space through training. However, the matching of event arguments related to breaking events on platforms like Weibo is problematic. Figure 1 Generally, it supplements textual information, unlike news images which satisfy its assumptions; 2) This invention introduces image information through image description generation, supplementing Weibo text information by describing the image content; 3) This invention models the task as a text sequence labeling task, which is more conducive to transferring background knowledge learned on large-scale text corpora by using pre-trained models, effectively alleviating the problem of difficult model training due to limited labeling resources; 4) This invention uses machine reading comprehension to effectively integrate the structural information of events into the input sequence, allowing the model to differentiate the modeling of various event arguments, thereby improving the argument extraction effect. Attached Figure Description

[0035] Figure 1 This is a general framework diagram of a burst argument extraction method for multimodal data based on machine reading comprehension.

[0036] Figure 2 This is a flowchart of a burst argument extraction method for multimodal data based on machine reading comprehension. Detailed Implementation

[0037] The implementation process of the present invention will be described in detail below with reference to the embodiments and accompanying drawings.

[0038] Example 1: Reference Figure 1 and Figure 2 The application is a machine reading comprehension based multi-modal data oriented burst argument extraction method, which comprises the following steps:

[0039] Step 1) The image features of the microblog image are extracted by using the ResNet model.

[0040] Step 2) The image features of the microblog image obtained in step 1) are decoded by the LSTM model to obtain the image description of the microblog image.

[0041] Step 3) The model in step 2) is trained in a large amount of annotated image-text data in a Teacher-Forcing manner, and in actual prediction, the beam search method is used to obtain the text description of the image.

[0042] Step 4) The event type, the argument to be extracted, the microblog text information and the microblog image description are spliced into a natural language chapter and a question.

[0043] Step 5) The chapter and the question obtained in step 4) are converted into corresponding vector representations by the pre-training model BERT.

[0044] Step 6) The required event argument is extracted from the vector representation obtained in step 5) by the machine reading comprehension model.

[0045] Step 7) The initial model is trained in a large amount of annotated event argument data by the mini-batch gradient descent method to obtain an optimized event argument extraction model.

[0046] In step 2), the decoder decodes the microblog image representation generated by the ResNet into the description of the image. The problem is modeled as a one-way recursive autoregressive model, so that the probability of the generated sequence on the language model is maximum. The specific implementation is as follows:

[0047]

[0048] Where X is a sequence set input into the model, X i is the i-th input word, Y is a sequence set output from the model, Y i is the i-th output word

[0049] In step 3), the model can predict the input and output of each step of the decoder in advance due to the labeled data pairs in the training stage, so the target sequence can be shifted by one bit for training, which is more convenient for training and the model is easier to learn the ability to generate corresponding sequences. This training method is called Teacher-Forcing. But this training method is only implemented in training, and does not exist in the prediction stage, so the decoder cannot use the input of the next time step in advance when performing each step.

[0050] In the actual prediction stage, the sequence with the maximum probability can be obtained by enumerating all possible output sequences, but the computational overhead is too large, and this is not a Markov process and cannot be solved using dynamic programming. If the output with the maximum probability is selected each time, it is called greedy search, which has much less computational overhead than enumerating all sequences in the past, but the results obtained by this method are often not optimal. Assuming that the first step selects a probability that is not the maximum, when the second step is entered, a very high conditional probability p(Y2|X, Y1) can be obtained, so the result of multiplying the two will exceed the maximum of the greedy algorithm. In order to efficiently and accurately solve this problem, a beam search strategy is used. For each generation process, first, set the hyperparameter k as the beam size of beam search. Second, in the first time step, select the k words with the maximum conditional probability in the language model at the current time step to form the first word of the k candidate output sequences. Third, in each subsequent time step, based on the k candidate output sequences of the previous time step, concatenate each word in the candidate dictionary to select the k sequences with the maximum conditional probability from the new possible output sequences as the candidate output sequences of this time step. Finally, filter the candidate output sequences from each time step to obtain the final output sequence <eos>" and all special symbols <eos>The subsequence after the " " is discarded, and a set of final candidate output sequences is obtained.

[0051] The step 4) splices the event type, the argument to be extracted, the microblog text information and the microblog image description into a natural language passage and a question. This step converts the extraction task into a machine reading comprehension task, and reasonable construction of the question pair model plays an important role in improving the extraction effect. The known event type and the argument to be extracted are packaged into a question using a template, and a special symbol [CLS] is added in front of the question; the microblog text and the question are spliced, and a special symbol [SEP] is used to separate them; the image description of the microblog image is spliced with the question and the text sequence, and a special token [IMG] is used to separate them; and the finally spliced sequence is:

[0052] query = [CLS] + query_pattern (event_type, argument) + [SEP] + sentence + [IMG] + image_caption

[0053]

[0054] [CLS], [SEP] and [IMG] are special markers, query_pattern is a question generation template, see Table 1 for details, event_type is an event type, argument is the name of an event argument to be extracted, sentence is a microblog text sequence, and image_caption is a text description generated by a microblog image.

[0055] As a preferred example, the question constructed in the step 4) has a guiding effect on the machine reading comprehension model, and the question constructed according to different templates has a related influence on the extraction effect. For a double-pointer argument extraction model, this paper attempts to extract event arguments using three different query patterns, as shown in Table 1. Template 1 directly splices the event type and the argument name, which clearly and accurately gives the event type and the event argument to be extracted; template 2 introduces the argument role description on the basis of template 1, which uses more common language to explain the argument in the vertical domain corpus, but the generated question is longer, which is more helpful for pre-training language model to transfer knowledge from unsupervised corpus. The description of each argument in the aircraft military exercise event is shown in Table 2; and template 3 additionally adds a description, which makes the guidance of the model to extract the specified argument more explicit.

[0056] Table 1: Event argument extraction template description table

[0057]

[0058] ​Table 2 Argument descriptions corresponding to different arguments in the aircraft military exercise event

[0059]

[0060] In step 5), the passage and question obtained in step 4) are converted into corresponding vector representations by the pre-trained model BERT, and the conversion is performed according to the following steps.

[0061] Step 51) generates Token Embedding, that is, word Embeddings. For Chinese, the words are segmented by word, and for English, the words are preprocessed in the form of WordPiece to split the words into morphemes, thereby reducing the influence of out-of-vocabulary words.

[0062] Step 52) generates Segment Embedding, which is used to distinguish each part spliced. The question area is marked as 0, the text token area is marked as 1, and the image token area is marked as 2, which can ensure that the extracted argument span exists in the text sequence.

[0063] Step 53) generates Position Embedding, which encodes the position of each input and adds position information, and uses the bit-by-bit encoding method of BERT.

[0064] Step 54) adds the above three embedding layers to obtain the Embedding layer.

[0065] Step 55) inputs the text Embedding layer vector into the standard BERT pre-training model to obtain the sequence feature representation.

[0066] In step 6), the required event argument is extracted from the vector representation obtained in step 5) by using a machine reading comprehension model. The specific method is to take out the vector at the [CLS] position, use a fully connected layer to convert the BERT output into a hidden layer with a dimension of 2, which respectively represents the start position and end position of the argument. The cross-entropy loss function is used to calculate the loss value between the real label and the average loss value after adding. Finally, by counting the span of the start position and the end position, the argument to be extracted can be extracted.

[0067] As a preferred example, Figure 1 A military exercise event is described, and the event argument to be extracted is the location of the military exercise. The required event argument is extracted from the vector representation obtained in step 5) by the machine reading comprehension model, and the location of the military exercise is South Korea. The method is to obtain the feature vector output by BERT bit by bit, and convert it into a hidden layer with a dimension of 2 using a fully connected layer. In turn, it is determined whether it is a start position or an end position, and finally the span of the start position and the end position is counted to extract the position of the argument to be extracted in the sequence.

[0068] In step 7), the traditional gradient descent is used, and the average gradient is calculated for all training data each time the gradient is descended. This gradient descent method is called full-batch gradient descent. Considering a case where the amount of training data is in the order of ten million, how long it takes to wait for one iteration greatly reduces the training speed. If a suitable bathsize data amount is selected for training, it is called mini-batch gradient descent.

[0069] The disadvantage of stochastic gradient descent is that the same data cannot be guaranteed to be used for each training, so each batch cannot be guaranteed to be descended. The overall training loss changes will have a lot of noise, but the overall trend is to decline, and then fluctuate around the optimal value, and will not converge.

[0070] During training, the accuracy p, recall r and balance index F1 are used to observe the training situation:

[0071]

[0072]

[0073]

[0074] To verify the effect of the method of the application, the method of the application is compared with some argument extraction methods of traditional single modal. Since there is no multi-modal event extraction data set about emergencies at present, a multi-modal event extraction data set about public military intelligence is constructed based on the microblog platform in this example. In this embodiment, a validation set is set in the data set, and the convergence of the current model is observed at the end of each training to prevent overfitting on the training set. Since the data on the validation set does not appear in the training, the effect of the model on the validation set can measure the generalization of the model.

[0075] This embodiment divides the entire labeled data according to 8:1:1, of which 80% is used for training, 10% for validation, and 10% for testing. The three sets have no intersection. After obtaining the training model, event extraction is performed on the data in the test set, and the indicators of argument extraction are evaluated respectively. The effect on this data set is shown in Table 3.

[0076] Model name Accuracy Recall F1 BERT 63.52％ 55.38％ 59.17％ BERT-CRF 66.10％ 58.48％ 62.06％ BERT-MRC 68.68％ 63.62％ 66.05％ Chinese-DeepNER 69.76％ 68.49％ 69.12％ The method of the present application 70.36％ 73.95％ 72.11％

[0077] BERT is a commonly used pre-training language model, and this embodiment also attempts to use other popular pre-training language models for comparison. The effect difference on the event argument extraction task is not obvious. Chinese-DeepNER uses the public implementation of the author on GitHub, and other models are implemented using bert4keras. The experimental results show that the method of the present application can effectively extract arguments for multi-modal emergency events, and the reasonable introduction of picture modal information makes the model improve the extraction accuracy. The F1 indicator shows that the existing pre-training model can well capture the relationship between the texts and extract high-quality text features. On this task, using a machine reading comprehension model can achieve better results than CRF.

[0078] It should be noted that the above embodiments are not intended to limit the scope of protection of the present application, and any equivalent transformation or substitution made on the basis of the above technical solutions falls within the scope of protection of the claims of the present application.< / eos> < / eos> < / eos> < / eos>

Claims

1. A method for event argument extraction oriented to multi-modal data, characterized in that, The method comprises the following steps: Step 1) extracting the image features of the microblog image by using the ResNet model; Step 2) decoding the image features of the microblog image obtained in step 1) by using the LSTM model to obtain the image description of the microblog image; Step 3) training the model in step 2) in a Teacher-Forcing manner on a large amount of labeled image-text data, and using beam search to obtain the text description of the image during actual prediction; Step 4) concatenating the event type, the argument to be extracted, the microblog text information and the microblog image description into a natural language passage and a question; Step 5) converting the passage and the question obtained in step 4) into corresponding vector representations by using the pre-training model BERT; Step 6) extracting the required event argument from the vector representations obtained in step 5) by using a machine reading comprehension model; Step 7) training the initial model by using the mini-batch gradient descent method on a large amount of labeled event argument data to obtain an optimized event argument extraction model; In step 2), the decoder decodes the microblog image representation generated by the ResNet into the description of the image, and models the question as a one-way recursive autoregressive model to maximize the probability of the generated sequence on the language model, which is specifically implemented as: ； where X is a set of sequences input to the model, X i is the i-th input word, Y is a set of sequences output from the model, Y i is the i-th output word; In step 5), the pre-training model BERT is used to convert the passage and the question obtained in step 4) into corresponding vector representations, which is converted according to the following steps, Step 51) generate Token Embedding, i.e. word Embeddings, for Chinese, use word segmentation, for English, use WordPiece preprocessing method to split words into morphemes, reduce the impact of out-of-vocabulary words; Step 52) generate Segment Embedding to distinguish between the parts spliced, mark the question area as 0, the text token area as 1, and the image token area as 2 to ensure that the extracted argument span exists in the text sequence; Step 53) generate Position Embedding to encode each input position and add position information, which uses the BERT bit-by-bit encoding method; Step 54) add the above three embedding layers to obtain the Embedding layer; Step 55) input the text Embedding layer vector into the standard BERT pre-training model to obtain the sequence feature representation.

2. The multi-modal data oriented event argument extraction method according to claim 1, wherein, In step 4), the event type, the argument to be extracted, the microblog text information and the microblog image description are concatenated into a natural language question and passage sequence.

3. The multi-modal data oriented event argument extraction method according to claim 2, characterized in that, In step 6) of machine reading comprehension, the question generation method is to wrap the known event type and the argument to be extracted into a question using a template, and add a special symbol [CLS] mark in front of the question.

4. The multi-modal data oriented event argument extraction method according to claim 3, characterized in that, Step 6) In the machine reading comprehension question, the generation method of the text sequence is to splice the image description of the micro-blog picture with the question and the text sequence, and use special tokens [IMG] to separate them.

5. The multi-modal data oriented event argument extraction method according to claim 4, characterized in that, Step 6) In the machine reading comprehension question, the splicing method of the question and the text sequence is to use special symbols [SEP] to separate them; the final spliced sequence is: ； [CLS], [SEP] and [IMG] are special markers, query_pattern is the template for generating questions, event_type is the event type, argument is the event argument to be extracted, sentence is the text sequence of the micro-blog, and image_caption is the text description generated by the micro-blog picture.

6. The multi-modal data oriented event argument extraction method according to claim 1, wherein, In step 6, the required event argument is extracted from the vector representation obtained in step 5) by the machine reading comprehension model.

7. The multi-modal data oriented event argument extraction method according to claim 6, characterized in that, Step 6) In the machine reading comprehension question, the vector at the [CLS] position is taken out, the output of BERT is converted to a hidden layer with a dimension of 2 using a fully connected layer, which represents the start position and end position of the argument respectively, the loss value between the calculated cross-entropy loss function and the true label is calculated respectively, the final loss value is obtained by adding and averaging, and finally the span of the start position and end position is counted, which can extract the argument to be extracted.

8. The multi-modal data oriented event argument extraction method according to claim 1, wherein, In step 7, the mini-batch gradient descent method is used to train the model, which can effectively help the model converge smoothly on large-scale data sets.

Images