Zero sample cross-language event argument extraction method based on mT5 model

Abstract

The invention relates to the technical field of text information extraction, in particular to a zero sample cross-language event argument extraction method based on an mT5 model. A semantic dependency analysis prefix is introduced, a semantic dependency relationship is included through a semantic-syntax double-prefix constructor, a language-independent double-prefix generator of a cross-language fusion syntax and semantic dependency is constructed according to a language-independent EAE template, a semantic-syntax double-prefix adjustment mT5 decoder state is obtained, a zero-sample cross-language event argument is obtained, and the zero-sample cross-language event argument is obtained. SRL and SDP are combined to generate a synergistic effect, semantic features are enriched, and the performance of an EAE system in a natural language processing task is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of text information extraction, and in particular to a zero-sample cross-language event argument extraction method based on an mT5 model. Background Art

[0002] In recent years, the emergence of large-scale language models (LLMs) has been reshaping the field of artificial intelligence. In particular, the most advanced LLMs, such as ChatGPT, GPT-4, etc., have billions of parameters and have gained considerable attention due to their outstanding performance in various natural language generation tasks. The release of many high-performance open source LLMs has also stimulated the public's enthusiasm for LLM research and application. For example, mT5 is a multilingual variant of the T5 model developed by Google. The training data set covers 101 languages ​​and contains 300 million to 13 billion parameters. As a multilingual LLM, it can share information between similar languages, reduce dependence on data and resources, and allow few-shot or zero-shot learning.

[0003] Understanding text by identifying events and arguments has been a long-standing research problem in the field of natural language processing. Since great progress has been made in event extraction, the main challenge of this work has shifted to event argument extraction (EAE). EAE aims to identify entities that serve as event arguments and determine their specific roles in the event. Figure 1 In the English example, the word "arrested" is used as the trigger word of the "Arrest" event. Event argument extraction will identify the event arguments as: "the criminal", "policemen" and "yesterday", and classify them into the following roles: "Target", "Arrester" and "Time" respectively. As an important branch of event extraction, EAE plays an important role in improving the effects of various downstream applications, such as event timeline generation, AI dialogue systems and recommendation systems.

[0004] However, the effectiveness of EAE models depends heavily on whether there is a large amount of valid available data. This requirement poses a challenge for limited-resource languages, as there is often a lack of sufficient annotated samples to train models on the target language.

[0005] To solve this problem, a cross-language EAE strategy can be considered. Cross-language EAE uses data-rich source languages ​​to enhance event argument extraction capabilities in resource-poor target languages, a practice that has received increasing attention in recent years. In addition, previous studies have shown that there are significant differences in the average syntactic distances between trigger words and potential arguments in different languages, with average distances of 9.8 and 21.7 for English and Chinese, respectively. Therefore, a model trained on English data may over-focus on words that are closer, affecting its ability to be effectively generalized to Chinese. Therefore, it is considered to advance important word units to the position that the model focuses on as prefixes.

[0006] The present invention is dedicated to the Zero-shot Cross-language EventArgument Extraction (ZCEAE) method. Figure 2 As shown in

[15] , ZCEAE refers to training the EAE model entirely with resource-rich source language data and then applying it to different target languages ​​without the need for any annotated data in the target language, but existing methods often ignore the necessity of semantic information. Summary of the invention

[0007] In view of the fact that existing text information extraction methods often ignore the necessity of semantic information and have insufficient performance in processing natural language processing tasks, the present invention proposes a zero-sample cross-language event argument extraction method based on the mT5 model; by introducing semantic dependency parsing prefixes, the semantic dependency relationship is incorporated through a semantic-syntactic double prefix constructor, and a language-independent double prefix generator that cross-language integrates syntax and semantic dependencies is constructed according to a language-independent EAE template, and the semantic-syntactic double prefix is ​​obtained to adjust the mT5 decoder state, and the zero-sample cross-language event argument is obtained. By combining SRL with SDP to produce a synergistic effect, the semantic features are enriched, and the performance of the EAE system in natural language processing tasks is improved.

[0008] The specific implementation contents of the present invention are as follows:

[0009] A zero-shot cross-language event argument extraction method based on the mT5 model specifically includes the following steps:

[0010] Step S1: construct a zero-shot cross-language event argument extraction task;

[0011] Step S2: construct a language-independent EAE template based on the zero-shot cross-language event argument extraction task;

[0012] Step S3: Building the mT5 encoder-decoder architecture based on the language-independent EAE template;

[0013] Step S4: construct an SRL template, adjust the SRL template to a language-independent EAE template format, and concatenate it to the corresponding EAE training dataset;

[0014] Step S5: constructing a language-independent double prefix generator that integrates syntactic and semantic dependencies across languages ​​based on the language-independent EAE template to obtain a semantic-syntactic double prefix;

[0015] Step S6: Adjust the mT5 decoder state according to the semantic-syntactic double prefix, and use the negative log-likelihood function as the loss function to train and infer zero-sample cross-language event arguments.

[0016] In order to better implement the present invention, further, the step S1 specifically includes the following steps:

[0017] Step S11: define the input sentence S, Among them, w v is the vth word;

[0018] Step S12: define the event type set as ε, and define the argument role set R corresponding to the event type set ε e ;

[0019] Step S13: Define event trigger word O according to event type e and input sentence S; e∈ε;

[0020] Step S14: Construct a zero-shot cross-language event argument extraction task based on the argument role r and the continuous text span s.

[0021] In order to better implement the present invention, further, the specific operation of step S2 is: calling the HTML tag template, according to the event type e, making a language-independent EAE template

[0022] In order to better implement the present invention, further, the step S3 specifically includes the following steps:

[0023] Step S31: Sentence S, EAE template Concatenate into input sequence x;

[0024] Step S32: construct an mT5 encoder according to the input sequence x to obtain a hidden layer representation of the mT5 encoder;

[0025] Step S33: input the hidden layer representation of the mT5 encoder into the mT5 decoder, and generate word y word by word in an autoregressive manner;

[0026] Step S34: Calculate the conditional probability of word y according to the past state of the mT5 decoder.

[0027] In order to better implement the present invention, further, the step S4 specifically includes the following steps:

[0028] Step S41: adding additional statistical information of the SRL;

[0029] Step S42: constructing an SRL template according to the added additional statistical information;

[0030] Step S43: Adjust the constructed SRL template to a language-independent EAE template format, and concatenate it to the corresponding EAE training data set.

[0031] In order to better implement the present invention, further, the step S41 specifically includes the following steps:

[0032] Step S411: Call FrameNet data to train a role label classifier, and apply the trained role label classifier to PropBank data;

[0033] Step S412: The tag with the highest predicted frequency is used as the role name, and a statistical method is called to obtain the relative distance between the role and the predicate.

[0034] In order to better implement the present invention, further, the step S42 specifically includes the following steps:

[0035] Step S421: Formulate an SRL template according to the added additional statistical information and in combination with the set connection sorting component;

[0036] Step S422: Based on the language annotation, a statistical method is called to determine the role name of the language in the SRL framework;

[0037] Step S423: calling the FrameNet classifier and running it on the multi-language pre-trained encoder;

[0038] Step S424: Take the role with the highest confidence as the label, exclude prepositions, and arrange the roles according to the AGR number.

[0039] In order to better implement the present invention, further, step S5 specifically includes the following steps:

[0040] Step S51: calling Stanza to analyze syntactic dependency, calling HanLP to analyze semantic dependency, and constructing a coding dependency structure;

[0041] Step S52: convert the syntactic and semantic dependency trees generated by Stanza and HanLP into undirected graphs, use the trigger word as the starting point of the syntactic dependency tree, and use the breadth-first search method to select a specific number of neighbor nodes to obtain a syntactic prefix matrix and a semantic prefix matrix respectively;

[0042] Step S53: The syntactic prefix matrix and the semantic prefix matrix are used as the initialization representation of the prefix and attached to the Transformer layer of the mT5 encoder and the mT5 decoder to construct a language-independent double prefix generator that integrates syntactic and semantic dependencies across languages ​​to obtain the semantic-syntactic double prefix.

[0043] In order to better implement the present invention, further, the step S51 specifically includes the following steps:

[0044] Step S511: calling Stanza to analyze syntactic dependencies, calling HanLP to analyze semantic dependencies, constructing a dependency tree and calculating the distances between word pairs;

[0045] Step S512: Generate a distance matrix based on the sentence with a length of K;

[0046] Step S513: calling the mT5 encoder to map the word unit to the cross-language semantic space, and using the Transformer as the structural encoder to obtain the mask matrix B;

[0047] Step S514: Integrate syntax and semantics into the mask matrix to obtain self-attention distribution;

[0048] Step S515: Correct the self-attention distribution according to the distance between word pairs to obtain a corrected self-attention matrix.

[0049] In order to better implement the present invention, further, the step S52 specifically includes the following steps:

[0050] Step S521: converting the syntactic and semantic dependency trees generated by Stanza and HanLP into undirected graphs;

[0051] Step S522: using the trigger word as the starting point of the syntactic dependency tree, and selecting a specific number of neighbor nodes using a breadth-first search method;

[0052] Step S523: construct a trainable matrix according to the selected word units to obtain a syntactic prefix matrix and a semantic prefix matrix.

[0053] The present invention has the following beneficial effects:

[0054] The present invention constructs a semantic role labeling resource generator, combines the training data set of the event argument extraction task with the classic data set of the semantic role labeling task, produces a synergistic effect to enrich the semantic features, and improves the performance of the EAE system in natural language processing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A schematic diagram of the process of processing English examples using the traditional method provided by the present invention.

[0056] Figure 2 A schematic flowchart of the traditional zero-sample cross-language event argument extraction method provided by the present invention.

[0057] Figure 3 Schematic diagram of the overall model architecture of the zero-sample cross-language event argument extraction method based on the mT5 model provided by the present invention.

[0058] Figure 4 Schematic diagram of the semantic-syntactic double prefix builder model provided by the present invention.

[0059] Figure 5 A schematic diagram of the algorithm flow of the semantic-syntactic driven framework provided by the present invention. DETAILED DESCRIPTION

[0060] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. It should be understood that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments, and therefore should not be regarded as limiting the scope of protection. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technical personnel in this field without making creative work are within the scope of protection of the present invention.

[0061] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "disposed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0062] Embodiment 1:

[0063] This embodiment proposes a zero-shot cross-language event argument extraction method based on the mT5 model, which specifically includes the following steps:

[0064] Step S1: Construct a zero-shot cross-language event argument extraction task.

[0065] Furthermore, the step S1 specifically includes the following steps:

[0066] Step S11: define the input sentence S, Among them, w v is the vth word;

[0067] Step S12: define the event type set as ε, and define the argument role set R corresponding to the event type set ε e ;

[0068] Step S13: Define event trigger word O according to event type e and input sentence S; e∈ε;

[0069] Step S14: Construct a zero-shot cross-language event argument extraction task based on the argument role r and the continuous text span s.

[0070] Step S2: Based on the zero-shot cross-language event argument extraction task, a language-independent EAE template is constructed.

[0071] Furthermore, the specific operation of step S2 is: calling the HTML tag template, and making a language-independent EAE template according to the event type e

[0072] Step S3: Building the mT5 encoder-decoder architecture based on the language-independent EAE template.

[0073] Furthermore, the step S3 specifically includes the following steps:

[0074] Step S31: Sentence S, EAE template Concatenate into input sequence x;

[0075] Step S32: construct an mT5 encoder according to the input sequence x to obtain a hidden layer representation of the mT5 encoder;

[0076] Step S33: input the hidden layer representation of the mT5 encoder into the mT5 decoder, and generate word y word by word in an autoregressive manner;

[0077] Step S34: Calculate the conditional probability of word y according to the past state of the mT5 decoder.

[0078] Step S4: construct an SRL template, adjust the SRL template to a language-independent EAE template format, and concatenate it to the corresponding EAE training dataset.

[0079] Furthermore, the step S4 specifically includes the following steps:

[0080] Step S41: adding additional statistical information of the SRL;

[0081] Furthermore, the step S41 specifically includes the following steps:

[0082] Step S411: Call FrameNet data to train a role label classifier, and apply the trained role label classifier to PropBank data;

[0083] Step S412: The tag with the highest predicted frequency is used as the role name, and a statistical method is called to obtain the relative distance between the role and the predicate.

[0084] Step S42: constructing an SRL template according to the added additional statistical information;

[0085] Furthermore, the step S42 specifically includes the following steps:

[0086] Step S421: Formulate an SRL template according to the added additional statistical information and in combination with the set connection sorting component;

[0087] Step S422: Based on the language annotation, a statistical method is called to determine the role name of the language in the SRL framework;

[0088] Step S423: calling the FrameNet classifier and running it on the multi-language pre-trained encoder;

[0089] Step S424: Take the role with the highest confidence as the label, exclude prepositions, and arrange the roles according to the AGR number.

[0090] Step S43: Adjust the constructed SRL template to a language-independent EAE template format, and concatenate it to the corresponding EAE training data set.

[0091] Step S5: According to the language-independent EAE template, a language-independent double prefix generator is constructed that integrates syntactic and semantic dependencies across languages ​​to obtain a semantic-syntactic double prefix.

[0092] Furthermore, the step S5 specifically includes the following steps:

[0093] Step S51: calling Stanza to analyze syntactic dependency, calling HanLP to analyze semantic dependency, and constructing a coding dependency structure;

[0094] Furthermore, the step S51 specifically includes the following steps:

[0095] Step S511: calling Stanza to analyze syntactic dependencies, calling HanLP to analyze semantic dependencies, constructing a dependency tree and calculating the distances between word pairs;

[0096] Step S512: Generate a distance matrix based on the sentence with a length of K;

[0097] Step S513: calling the mT5 encoder to map the word unit to the cross-language semantic space, and using the Transformer as the structural encoder to obtain the mask matrix B;

[0098] Step S514: Integrate syntax and semantics into the mask matrix to obtain self-attention distribution;

[0099] Step S515: Correct the self-attention distribution according to the distance between word pairs to obtain a corrected self-attention matrix.

[0100] Step S52: convert the syntactic and semantic dependency trees generated by Stanza and HanLP into undirected graphs, use the trigger word as the starting point of the syntactic dependency tree, and use the breadth-first search method to select a specific number of neighbor nodes to obtain a syntactic prefix matrix and a semantic prefix matrix respectively;

[0101] Furthermore, the step S52 specifically includes the following steps:

[0102] Step S521: converting the syntactic and semantic dependency trees generated by Stanza and HanLP into undirected graphs;

[0103] Step S522: using the trigger word as the starting point of the syntactic dependency tree, and selecting a specific number of neighbor nodes using a breadth-first search method;

[0104] Step S523: construct a trainable matrix according to the selected word units to obtain a syntactic prefix matrix and a semantic prefix matrix.

[0105] Step S53: The syntactic prefix matrix and the semantic prefix matrix are used as the initialization representation of the prefix and attached to the Transformer layer of the mT5 encoder and the mT5 decoder to construct a language-independent double prefix generator that integrates syntactic and semantic dependencies across languages ​​to obtain the semantic-syntactic double prefix.

[0106] Step S6: Adjust the mT5 decoder state according to the semantic-syntactic double prefix, and use the negative log-likelihood function as the loss function to train and infer zero-sample cross-language event arguments.

[0107] Working principle: This embodiment first constructs an SRL resource generator, including making SRL templates and language-independent EAE templates, and processes the SRL data set into the form of an EAE data set and splices them together; secondly, Stanza is used for syntactic dependency analysis, and HanLP is used for semantic dependency analysis to build a dependency tree and calculate the distance between word pairs, thereby encoding the dependency structure; after encoding, this embodiment converts the syntactic and semantic dependency trees generated by Stanza and HanLP into undirected graphs, starting from the trigger word as the starting point of the syntactic dependency tree, and adopts the breadth-first search method to select a specific number of neighbor nodes to obtain the syntactic and semantic prefix matrices respectively. The matrix is ​​used as the initialization representation of the prefix and is attached to the sequence x and y at each Transformer layer of the mT5 encoder and decoder; then for the i-th word, the prefix U is associated with the previous word yi-1 Combined with the mT5 decoder status e i d Generation; the calculation of the mT5 encoder state follows a similar process; and finally training and inference. This embodiment finds that semantic information can also serve as an additional valuable resource for the zero-shot cross-language event argument extraction task like syntactic information. To this end, this embodiment proposes a framework called "semantic-syntactic driven" to fuse syntactic and semantic information. This embodiment designs a semantic role labeling resource generator that combines the training dataset of the event argument extraction task with the classic dataset of the semantic role labeling task in the form of a processing cost embodiment design. In addition, this embodiment also introduces a semantic dependency parsing prefix to incorporate semantic dependencies through a semantic-syntactic double prefix constructor. The method of this embodiment is significantly superior to previous advanced methods, and the average F1 scores on both the ACE-2005 and ERE datasets have been improved.

[0108] Embodiment 2:

[0109] This embodiment is based on the above embodiment 1. Figure 3 , Figure 4 , Figure 5 As shown, a specific embodiment is described in detail.

[0110] There are two main types of research on using semantic information to solve problems in this embodiment: Semantic Role Labeling (SRL) and Semantic Dependency Parsing (SDP). The combination of SRL and SDP can produce a synergistic effect, thereby enriching semantic features and improving the performance of the EAE system in natural language processing tasks.

[0111] Step S1: Construct a zero-shot cross-language event argument extraction task.

[0112] First, define an input sentence words, namely Among them, w v is the vth word.

[0113] The set of event types is defined as ε. For each event type e∈ε, define the corresponding argument role set R e .

[0114] Given an input sentence and an event trigger word O belonging to event type e∈ε, the event argument extraction task aims to identify all the event Such a tuple, where is an argument role of an event e, is the continuous span of text in a sentence.

[0115] For zero-shot cross-lingual event argument extraction, the source language training data used to train the EAE model is defined as:

[0116]

[0117] The target language test data is defined as:

[0118]

[0119] Among them, q represents the data sequence number, m and n represent the total amount of source language training data and target language test data respectively. Subsequently, the trained EAE model is directly tested on the examples of the target language.

[0120] Step S2: Create a language-independent EAE template.

[0121] Use HTML tag templates, each event type e corresponds to a template For example, the Arrest event includes four roles: Target, Arrester, Time, and Place. The template structure is: <target> [None]< / target> <arrester> [None]< / arrester> <time> [None]< / time>

[0122] <place> [None]< / place> ." Among them, the special token [None] without any linguistic association functions as a placeholder for arguments. The remaining special tokens such as 〈Target〉 and 〈 / Target〉 are still unknown to the pre-trained language model during the pre-training stage.

[0123] Furthermore, the placeholder [None] in the template is replaced with the corresponding truth value argument. If a role in the input sentence does not have a corresponding argument role, the placeholder [None] in the template is retained. For example, the output template of the above Attack event is " <target> The criminal< / target> <arrester> policeman< / arrester> <time> yesterday< / time> <place> [None]< / place> .” where “The criminal”, “policeman”, “yesterday” and [None] are real arguments.

[0124] In addition, if the same role prediction gets multiple arguments, they are preliminarily sorted according to the span and then connected together with the special symbol [and].

[0125] Step S3: Build the mT5 encoder-decoder architecture.

[0126] Take an input sentence template According to the method described in step S2, the input sequence is encoded and the corresponding representation is obtained as follows:

[0127]

[0128] The Encoder(·) function is an mT5 encoder. x represents the input sequence, which consists of the input sentence and templates [M] represents a separator token in the pre-trained language model. In addition, [:] represents an operator for concatenating sequences. x ∈R |x|×d Represents the hidden layer representation of each word in the corresponding input sequence.

[0129] Furthermore, the hidden layer representation is input into the mT5 decoder to generate the output sequence y word by word in an autoregressive manner. The mT5 decoder generates the i-th word y through the following process i and decoding status

[0130]

[0131] The Decoder(·) function represents an mT5 decoder.

[0132] Then, during decoding, the conditional probability p(y|x} of the entire output sequence can be calculated using the past state of the mT5 decoder calculate:

[0133]

[0134] Among them, p(y i |y} <i , x is the predicted word y i Given a token y generated in the past <i and the probability of input x.

[0135] Step S4: construct an SRL resource generator, that is, make an SRL template and a language-independent EAE template, so as to facilitate processing the SRL data set into the form of an EAE data set and splicing them together.

[0136] The main semantic role labeling resources of this embodiment include PropBank, NomBank and FrameNet. By checking the "source" attribute in the NomBank frame, it is mapped to the corresponding PropBank frame. Frames that lack such mapping will not be considered. By filtering the semantic role labeling frames related to the event, according to the POS set provided in the framework file, those frames that lack any verb implementation are excluded. In addition, only the non-core or modifier role subsets related to the target event argument extraction task are focused on. These roles include ARGM-LOC in PropBank and roles such as PLACE, INSTRUMENT, WEAPON, VEHICLE, etc. The specific construction method is as follows:

[0137] Step S41: Add additional statistical information of the SRL to help build the template.

[0138] The role label names of FrameNet are adopted. In addition, the role label classifier is trained using FrameNet data, and then the classifier is applied to the PropBank data, and the label with the highest frequency of the role name predicted by the classifier is used as its role name. For example, in the "buy.01" frame, the argument of the ARG0 role is mostly predicted as the BUYER label in the dataset, so "BUYER" is used as its role name.

[0139] A statistical method is used to collect the relative distance of each role from the predicate. For example, in the "buy.01" frame, for the instance "He bought a book in a store", the distance of ARG0(He) is -1, the distance of ARG1(book) is +1, and the distance of ARG-LOC(store) is +1. Subsequently, the order of roles in the template is determined based on the average relative distance of the roles. Only frame instances realized by non-passive verb predicates are considered.

[0140] For prepositional phrases, count the prevalence of using the argument of the preposition as a role (the criterion is: the dependency relationship between the head of the argument and the subword with POS "ADP" is "case"), and only keep the prepositions with a frequency of more than 25%. If such a preposition exists, put it before the role name and integrate it into the corresponding slot.

[0141] Step S42: construct an SRL template.

[0142] Using the above supplementary information, a template is formulated by connecting components with a determined role order. For example, the "buy.01" frame of PropBank is represented as a template "buyer buy goods for recipient from seller formoney in place.".

[0143] Furthermore, using Arabic and Chinese PropBank annotations from OntoNotes, we adopt the same statistical approach to determine the character names in the SRL frames for Arabic and Chinese: using the FrameNet classifier, which is run on a multilingual pre-trained encoder; for each character, the one with the highest confidence is used as the label; excluding prepositions, the characters are arranged according to their ARG numbers.

[0144] Step S43: adjusting the above SRL template to the language-independent EAE template format described in step S2, and concatenating it to the end of the corresponding EAE training data set.

[0145] Step S5: construct a language-independent double prefix generator that can fuse syntactic and semantic dependencies across languages. The specific construction method is as follows:

[0146] Step S51: Encode dependency structure: Use Stanza for syntactic dependency analysis and HanLP for semantic dependency analysis to build a dependency tree and calculate the distance between word pairs. This process generates a distance matrix for the input sentence (length K):

[0147] H∈R |K|×|K|

[0148] A hyperparameter α determines the distribution of the lengths of syntactic (α) and semantic (L-α) prefixes in the total prefix length L, where H ij Represents the syntactic (if 0≤i≤α-1) or semantic (if α≤i≤L-1) distance between the i-th token and the j-th token in the input sentence. The mT5 encoder is used to map the tokens into a cross-lingual semantic space, and then the Transformer is used as the structural encoder. The tokens pay attention to the rest of the tokens within a distance δ, which is represented by a mask matrix B:

[0149]

[0150] Among them, δ is a hyperparameter. This method can integrate syntactic and semantic structures into the mask matrix. For the lth layer of the Transformer encoder, the self-attention distribution P l Calculate as follows:

[0151]

[0152] where Q, K and V are the query, key and value of the lth layer of the Transformer encoder respectively, and the hidden layer size is d k .P l The element in row i and column j of It represents the attention assigned by the i-th word to the j-th word in the sentence.

[0153] Incorporating syntactic and semantic distances to modify self-attention distribution:

[0154]

[0155] in, is the regularization factor. A l is the modified self-attention matrix, A l The element in the i-th row and j-th column of is It will be used to calculate the output of the lth layer Transformer.

[0156] Step S52: Select neighbors: Convert the syntactic and semantic dependency trees generated by Stanza and HanLP into undirected graphs, starting from the trigger word as the starting point of the syntactic dependency tree, and use the breadth-first search method to select a specific number of neighbor nodes, such as Figure 4 Similarly, the neighbors of the semantic dependency tree are also selected through breadth-first search.

[0157] The selected words together constitute the trainable matrix U∈R L×d , where L represents the number of selected words. sy and U se denote the syntactic and semantic prefix matrices, respectively, which are used as the initialization representation of the prefix and are appended to the sequences x and y at each Transformer layer of the mT5 encoder and decoder.

[0158] Combine the prefix U with the previous word y i-1 , by introducing the prefix matrix U in the calculation, (2) where the mT5 decoder state The calculation is adjusted to:

[0159]

[0160] Among them, Decoder(·) is an mT5 decoder. It is the past state of the mT5 decoder during the decoding process. x ∈R |x|×d Represents the hidden representation corresponding to each token in the input sequence.

[0161] Step S6: Training and inference: The trainable parameters are collectively referred to as ξ. The negative log-likelihood function is used as the loss function:

[0162]

[0163] Among them, Tr is the training set of the source language, including both the data of semantic role labeling and the data of event argument extraction. c is the cth instance of the input sequence, G c is the corresponding true output sequence.

[0164] The model is trained based on the concatenated EAE dataset and SRL dataset in the source language obtained in step S4, namely, the dataset Tr. The training goal is to minimize the gap between the prediction and the true argument, using negative log-likelihood as the loss function, and finally predicting the target language data based on the trained model.

[0165] In general, this embodiment finds that semantic information can also be used as an additional valuable resource for the zero-shot cross-language event argument extraction task like syntactic information. To this end, this embodiment designs a "semantic-syntactic driven framework" to fuse semantic and syntactic information. Specifically, this embodiment designs a semantic role labeling resource generator that combines the training data set of the event argument extraction task with the classic data set of the semantic role labeling task in the form designed by the processing cost embodiment.

[0166] This embodiment also introduces semantic dependency parsing prefixes, and incorporates semantic dependency relationships through a semantic-syntactic double prefix builder. Experimental results show that this embodiment is significantly better than previous advanced methods, and the average F1 scores on the ACE-2005 and ERE datasets are both improved compared to the most advanced ZSEAE model.

[0167] The other parts of this embodiment are the same as those of the above-mentioned embodiment 1, and thus will not be described in detail.

[0168] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification or equivalent change made to the above embodiment based on the technical essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A zero-shot cross-language event argument extraction method based on the mT5 model, characterized in that: The specific steps include: Step S1: construct a zero-shot cross-language event argument extraction task; Step S2: construct a language-independent EAE template based on the zero-shot cross-language event argument extraction task; Step S3: Building an encoder-decoder architecture based on the language-independent EAE template and mT5 model; Step S4: construct an SRL template according to the added statistical information, adjust the SRL template to a language-independent EAE template format, and concatenate it to the corresponding EAE training data set; Step S5: constructing a language-independent double prefix generator that integrates syntactic and semantic dependencies across languages ​​based on the language-independent EAE template to obtain a semantic-syntactic double prefix; Step S6: Adjust the mT5 decoder state according to the semantic-syntactic double prefix, and use the negative log-likelihood function as the loss function to train and infer zero-sample cross-language event arguments.

2. According to the method for extracting zero-shot cross-language event arguments based on the mT5 model in claim 1, it is characterized in that: The step S1 specifically includes the following steps: Step S11: define the input sentence S, Step S12: define the event type set as ε, and define the argument role set R corresponding to the event type set ε e ; Step S13: Define event trigger word O according to event type e and input sentence S; e∈ε; Step S14: Construct a zero-shot cross-language event argument extraction task based on the argument role r and the continuous text span s.

3. According to claim 2, a zero-sample cross-language event argument extraction method based on the mT5 model is characterized in that: The specific operation of step S2 is: calling the HTML tag template, and making a language-independent EAE template according to the event type e 4. The method for extracting zero-shot cross-language event arguments based on the mT5 model according to claim 2, characterized in that: The step S3 specifically comprises the following steps: Step S31: Sentence S, EAE template Concatenate into input sequence x; Step S32: construct an mT5 encoder according to the input sequence x to obtain a hidden layer representation of the mT5 encoder; Step S33: input the hidden layer representation of the mT5 encoder into the mT5 decoder, and generate word y word by word in an autoregressive manner; Step S34: Calculate the conditional probability of word y according to the past state of the mT5 decoder.

5. According to claim 4, a zero-sample cross-language event argument extraction method based on the mT5 model is characterized in that: The step S4 specifically comprises the following steps: Step S41: adding statistical information of SRL; Step S42: constructing an SRL template according to the added statistical information; Step S43: Adjust the constructed SRL template to a language-independent EAE template format, and concatenate it to the corresponding EAE training data set.

6. The method for extracting zero-shot cross-language event arguments based on the mT5 model according to claim 5, characterized in that: The step S41 specifically includes the following steps: Step S411: Call FrameNet data to train a role label classifier, and apply the trained role label classifier to PropBank data; Step S412: The tag with the highest predicted frequency is used as the role name, and a statistical method is called to obtain the relative distance between the role and the predicate.

7. The method for extracting zero-shot cross-language event arguments based on the mT5 model according to claim 6, characterized in that: The step S42 specifically includes the following steps: Step S421: Formulate an SRL template based on the added statistical information and in combination with the set connection sorting component; Step S422: Based on the language annotation, a statistical method is called to determine the role name of the language in the SRL framework; Step S423: calling the FrameNet classifier and running it on the multi-language pre-trained encoder; Step S424: Take the role with the highest confidence as the label, exclude prepositions, and arrange the roles according to the AGR number.

8. The zero-sample cross-language event argument extraction method based on the mT5 model according to claim 7, characterized in that: The step S5 specifically comprises the following steps: Step S51: calling Stanza to analyze syntactic dependency, calling HanLP to analyze semantic dependency, and constructing a coding dependency structure; Step S52: convert the syntactic and semantic dependency trees generated by Stanza and HanLP into undirected graphs, use the trigger word as the starting point of the syntactic dependency tree, and use the breadth-first search method to select a specific number of neighbor nodes to obtain a syntactic prefix matrix and a semantic prefix matrix respectively; Step S53: The syntactic prefix matrix and the semantic prefix matrix are used as the initialization representation of the prefix and attached to the Transformer layer of the mT5 encoder and the mT5 decoder to construct a language-independent double prefix generator that integrates syntactic and semantic dependencies across languages ​​to obtain a syntactic-semantic double prefix.

9. The method for extracting zero-shot cross-language event arguments based on the mT5 model according to claim 8, characterized in that: The step S51 specifically includes the following steps: Step S511: calling Stanza to analyze syntactic dependencies, calling HanLP to analyze semantic dependencies, constructing a dependency tree and calculating the distances between word pairs; Step S512: Generate a distance matrix based on the sentence with a length of K; Step S513: calling the mT5 encoder to map the word unit to the cross-language semantic space, and using the Transformer as the structural encoder to obtain the mask matrix B; Step S514: Integrate syntax and semantics into the mask matrix to obtain self-attention distribution; Step S515: Correct the self-attention distribution according to the distance between word pairs to obtain a corrected self-attention matrix.

10. The zero-shot cross-language event argument extraction method based on the mT5 model according to claim 9, characterized in that: The step S52 specifically includes the following steps: Step S521: converting the syntactic and semantic dependency trees generated by Stanza and HanLP into undirected graphs; Step S522: using the trigger word as the starting point of the syntactic dependency tree, and selecting a specific number of neighbor nodes using a breadth-first search method; Step S523: construct a trainable matrix according to the selected word units to obtain a syntactic prefix matrix and a semantic prefix matrix.

Images