Generative event relation extraction method and system based on exogenous knowledge enhancement

Abstract

The invention provides a generative event relation extraction method and system based on exogenous knowledge enhancement, and relates to the field of natural language processing. The method comprises the following steps: acquiring to-be-analyzed text data, and inputting the text data into a pre-training language model ERNIE-GEN, so that the pre-training language model ERNIE-GEN is combined with a common sense knowledge base ATOMIC2020 in a fine tuning manner and common sense knowledge related to an event is fused into the model; based on the pre-training language model ERNIE-GEN, converting an event relationship extraction task into a generative task, and generating a time relationship between the at least two events; in a generative task, optimizing encoder parameters of the pre-training language model ERNIE-GEN through an auxiliary classification task; and outputting the time relationship between the at least two events. A better effect is achieved in various application scenes such as event relation extraction and time sequence knowledge graph construction.

Description

Technical Field

[0001] The present application relates to the field of natural language processing, and in particular to a generative event relationship extraction method and system based on exogenous knowledge enhancement. Background Art

[0002] Event relationship extraction technology refers to obtaining the temporal relationship between events in natural language text, such as determining whether an event occurs before, after, or at the same time as another event. Existing research also mentions inclusion and inclusion as well as uncertain temporal relationships, that is, events will have a duration, and then there will be inclusion and inclusion relationships, as well as temporal sequences that cannot be analyzed in context, that is, uncertain relationships. Relationship extraction between events can help understand the dynamics of complex events in natural language, which is beneficial to various downstream tasks, including the construction of event graphs, the analysis and construction of temporal knowledge graphs, the prediction of future events, and question-answering and summarization, information retrieval tasks, and the timeline of events can be sorted out by combining the relationship between events. By understanding the temporal relationship of events that have occurred, constructing event graphs and event-related temporal knowledge graphs, it is possible to predict future events, and construct event timelines based on the constructed sequence, thereby improving the text quality in generation tasks. Existing research on this task mainly includes the following methods: converting the problem into a multi-label classification task, using an encoder to obtain the representation of event pairs, and then inputting them into a multi-layer perceptron for prediction. This method also includes research on the representation of event pairs. Better representation can improve the classification effect, as well as the construction of classification models, including the adjustment of features and attention, so that it is more suitable for the task of analyzing the temporal relationship between events and achieves better classification results, such as the application of pre-trained language models to improve the representation of events; other research methods build graph models to learn the semantic connection between different event trigger words, or use different attention mechanisms to focus on the content between events and before and after events, to focus on event trigger words and the text fragments in between. , allowing for a more fine-grained consideration of the relationship between sentence features and events, thereby achieving relational classification; other studies consider the introduction of external knowledge. The use of common sense knowledge in the task of extracting event relations based on end-to-end neural architectures is quite limited. Especially in some common sense issues, the model is easily misled by the semantics of the context. Therefore, in the face of this problem, it is necessary to explore the integration of external knowledge to alleviate the scarcity of event annotations, thereby improving the effect of extracting event relations between events; other studies convert temporal relations into logical expressions of time points, and complete the task by predicting the relationship between certain time points, using it as a unified event-time relationship extraction framework. This method is more fine-grained and can better highlight the in-depth understanding of the relationship between events. The rapid development of large model technology has also enabled it to show good results in the task of extracting event-time relations. By combining the learning paradigm of retrieval enhancement and prompt learning, and using the rich pre-training knowledge of large models to analyze the temporal order relationship between events, longer context input can make the analysis process more holistic, thereby obtaining more comprehensive event relationship analysis results.

[0003] Common sense knowledge refers to exogenous common sense-related knowledge that is independent of training data. For text reading, humans make common sense inferences to construct their understanding of the narrative presented. For machines to achieve this ability, they must be able to obtain relevant and correct common sense in unlimited situations. Common sense knowledge is very important for downstream tasks of natural language processing, and is particularly critical for the event relationship extraction task targeted by this application, because language expression is always rich and diverse, and the limited length of text will omit a lot of common sense knowledge. The diversity of language expression of time relations in different contexts. Although there are some similarities in language, there is some common but unspoken knowledge in language expression, which determines the way time information is expressed. For example, "graduation" is generally before "employment" and "eating" is after "cooking". For example, in medical care, preventive measures are widely taken, and many treatments are only effective before illness. In contrast, in the repair equipment industry, it is common for most people to repair their equipment after a problem occurs. However, although it is easy to understand for human knowledge reserves, this common sense knowledge is rarely explicitly stated in texts and varies greatly in different fields, so common sense knowledge is needed as prior knowledge to improve the model's common sense knowledge reserves. Existing technologies all rely on pre-trained language models to improve the semantic understanding of context and events in samples, and then realize the analysis of the relationship between events, but the knowledge of the pre-trained language model comes from a large amount of corpus in the pre-training stage and will be affected by the distribution of facts in the pre-training stage. It is difficult to fully express the knowledge in the pre-trained language model by relying solely on downstream tasks to train event relationship analysis tasks. Knowledge graphs can help language models better transfer as knowledge engines by retraining language models on examples of structured knowledge, and help language models generalize to hypothetical knowledge tuples.

[0004] Therefore, how to improve the generalization ability of the model on specific downstream tasks and reduce the performance gap between pre-training tasks and downstream tasks has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] In order to improve the generalization ability of the model on specific downstream tasks and reduce the performance difference between pre-training tasks and downstream tasks, the present application provides a generative event relationship extraction method and system based on exogenous knowledge enhancement.

[0006] In the first aspect, the present application provides a generative event relationship extraction method based on exogenous knowledge enhancement, which adopts the following technical solution: A generative event relation extraction method based on exogenous knowledge enhancement, comprising: Acquire text data to be analyzed, wherein the text data contains at least two events; Input the text data into the pre-trained language model ERNIE-GEN, so that the pre-trained language model ERNIE-GEN is combined with the common sense knowledge base ATOMIC2020 through fine-tuning and the common sense knowledge related to the event is integrated into the model; Based on the pre-trained language model ERNIE-GEN, converting the event relationship extraction task into a generative task to generate the temporal relationship between the at least two events; In the generative task, the encoder parameters of the pre-trained language model ERNIE-GEN are optimized through the auxiliary classification task; The time relationship between the at least two events is output, wherein the time relationship includes at least one of a sequence of occurrence of the events, synchronous occurrence, inclusion relationship, or uncertain relationship.

[0007] Optionally, the step of inputting the text data into a pre-trained language model ERNIE-GEN so that the pre-trained language model ERNIE-GEN is combined with the common sense knowledge base ATOMIC2020 by fine-tuning and integrating the common sense knowledge related to the event into the model includes: Input the text data into the pre-trained language model ERNIE-GEN so that the pre-trained language model ERNIE-GEN constructs data in the common sense knowledge base ATOMIC2020 and performs training; Incorporate factual knowledge from the trained common sense knowledge base ATOMIC2020 into the pre-trained language model ERNIE-GEN.

[0008] Optionally, the data in the common sense knowledge base is triple data, and the model is trained by a sequence-to-sequence generation method so that the model can generate missing head entities, relations or tail entities.

[0009] Optionally, based on the pre-trained language model ERNIE-GEN, converting the event relationship extraction task into a generative task to generate the temporal relationship between the at least two events comprises: The data set is adjusted, and the source sequence and target sequence are constructed based on the text Text to be analyzed in the existing data set, the two events Event1 and Event2 to be analyzed, the label data Label and the prompt text: use Encode and obtain sequence representation: Among them, X and Y are the Token serialization representations of the source text Source and the target text Target respectively; The Token representation of the encoded source text and target text is input into the encoder and decoder parts of the model as the source sequence and target sequence: Among them, the embedding vector corresponding to the encoder is expressed as , represents the embedding vector of the i-th Token, n is the number of Tokens in the source text, and the embedding vector corresponding to the decoder is expressed as , represents the embedding vector of the jth Token in the target sequence, and m is the number of Tokens in the target text; in, represents the hidden state of the encoder at each time step; The decoder accepts the encoder's hidden state and the embedding vector of the decoder input , calculate the decoder's hidden state and output probability distribution: in, is the hidden state of the decoder at time step t-1, and is the weight matrix and bias vector of the output layer, is the output probability distribution of time step t, generating the token of time step t according to the input source sequence and the representation of the t-1 step before time step t, and the loss is calculated using the following loss: in, is the length of the target sequence.

[0010] Optionally, in the generative task, the step of optimizing the encoder parameters of the pre-trained language model ERNIE-GEN through an auxiliary classification task includes: The classification task is used as an auxiliary task and jointly trained with text generation to obtain the encoded text representation: in, represents the hidden state of the encoder at each time step; Use the last hidden state output by the encoder as the context representation of the entire input sequence: A multi-layer perceptron (MLP) is constructed on the encoder output as a classifier. The classifier consists of a fully connected layer and a softmax layer. The fully connected layer represents the context. Mapping to category scores : in, and Represent the weight matrix and bias vector of the classifier respectively; The Softmax layer converts the category scores Convert to category probability distribution p: This task uses cross entropy loss to calculate the predicted category distribution p and the true category label The difference between: in, is the number of categories, is the one-hot representation of the true category label, is the probability of the predicted category; The overall training loss during the training phase can be expressed as: in, and As a hyperparameter, it needs to be adjusted based on the training effect.

[0011] In a second aspect, the present application provides a generative event relationship extraction system based on exogenous knowledge enhancement, comprising: A data acquisition module, used to acquire text data to be analyzed, wherein the text data contains at least two events; A fine-tuning module, used for inputting the text data into the pre-trained language model ERNIE-GEN, so that the pre-trained language model ERNIE-GEN is combined with the common sense knowledge base ATOMIC2020 through fine-tuning and the common sense knowledge related to the event is integrated into the model; A generation module, configured to convert the event relationship extraction task into a generation task based on the pre-trained language model ERNIE-GEN, and generate a temporal relationship between the at least two events; An auxiliary module, used for optimizing the encoder parameters of the pre-trained language model ERNIE-GEN through an auxiliary classification task in a generative task; The output module is used to output the time relationship between the at least two events, where the time relationship includes at least one of the sequence of event occurrence, synchronous occurrence, inclusion relationship or uncertain relationship.

[0012] Optionally, the fine-tuning module is further used to input the text data into the pre-trained language model ERNIE-GEN so that the pre-trained language model ERNIE-GEN constructs data in the common sense knowledge base ATOMIC2020 and performs training; Incorporate factual knowledge from the trained common sense knowledge base ATOMIC2020 into the pre-trained language model ERNIE-GEN.

[0013] Optionally, the generating module is further used to adjust the data set, and construct the source sequence and the target sequence based on the text Text to be analyzed in the existing data set, two events Event1 and Event2 to be analyzed, label data Label and the prompt text as follows: use Encode and obtain sequence representation: Among them, X and Y are the Token serialization representations of the source text Source and the target text Target respectively; The Token representation of the encoded source text and target text is input into the encoder and decoder parts of the model as the source sequence and target sequence: Among them, the embedding vector corresponding to the encoder is expressed as , represents the embedding vector of the i-th Token, n is the number of Tokens in the source text, and the embedding vector corresponding to the decoder is expressed as , represents the embedding vector of the jth Token in the target sequence, and m is the number of Tokens in the target text; in, represents the hidden state of the encoder at each time step; The decoder accepts the encoder's hidden state and the embedding vector of the decoder input , calculate the decoder's hidden state and output probability distribution: in, is the hidden state of the decoder at time step t-1, and is the weight matrix and bias vector of the output layer, is the output probability distribution of time step t, generating the token of time step t according to the input source sequence and the representation of the t-1 step before time step t, and the loss is calculated using the following loss: in, is the length of the target sequence.

[0014] In a third aspect, the present application provides a computer device, comprising: a memory and a processor, wherein the processor executes the method described above when running computer instructions stored in the memory.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium, comprising instructions, which, when executed on a computer, enable the computer to execute the method described above.

[0016] In summary, the present application includes the following beneficial technical effects: This application obtains text data to be analyzed, and inputs the text data into the pre-trained language model ERNIE-GEN, so that the pre-trained language model ERNIE-GEN is combined with the common sense knowledge base ATOMIC2020 through fine-tuning and the common sense knowledge related to the event is integrated into the model; Based on the pre-trained language model ERNIE-GEN, the event relationship extraction task is converted into a generative task to generate the time relationship between the at least two events; In the generative task, the encoder parameters of the pre-trained language model ERNIE-GEN are optimized through auxiliary classification tasks; The time relationship between the at least two events is output. By combining pre-trained language models, knowledge enhancement, prompt learning and multi-task learning, as well as common sense knowledge bases and generative tasks, better results are achieved in various application scenarios such as event relationship extraction and time series knowledge graph construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of the computer device structure of the hardware operating environment involved in the embodiment of the present application; Figure 2 It is a flowchart of the first embodiment of the generative event relationship extraction method based on exogenous knowledge enhancement of the present application; Figure 3 This is a conceptual diagram of the stages of the first embodiment of the generative event relationship extraction method based on exogenous knowledge enhancement of the present application; Figure 4 It is a structural block diagram of the first embodiment of the generative event relationship extraction system based on exogenous knowledge enhancement of the present application. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below through the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0019] Reference Figure 1 , Figure 1 A schematic diagram of the computer device structure of the hardware operating environment involved in the embodiment of the present application.

[0020] like Figure 1 As shown, the computer device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM), or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk storage. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0021] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the computer device, and may include more or less components than shown in the figure, or combine some components, or arrange the components differently.

[0022] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a generative event relationship extraction program based on external knowledge enhancement.

[0023] exist Figure 1In the computer device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the present application can be set in the computer device, and the computer device calls the generative event relationship extraction program based on exogenous knowledge enhancement stored in the memory 1005 through the processor 1001, and executes the generative event relationship extraction method based on exogenous knowledge enhancement provided in the embodiment of the present application.

[0024] The present application embodiment provides a generative event relationship extraction method based on exogenous knowledge enhancement, referring to Figure 2 , Figure 2 This is a flow chart of the first embodiment of the generative event relationship extraction method based on exogenous knowledge enhancement of the present application.

[0025] In this embodiment, the generative event relationship extraction method based on exogenous knowledge enhancement includes the following steps: Step S10: Acquire text data to be analyzed, where the text data contains at least two events.

[0026] It should be noted that if Figure 3 The extraction process shown in the figure, the entire technology of this embodiment is divided into two stages. The first stage is the stage of integrating the common sense knowledge base into the model. In this stage, the model ERNIE-GEN is fine-tuned by means of the existing common sense knowledge base and the generative task, and the common sense knowledge related to the event is integrated into the model, and the common sense knowledge of event relationship analysis is provided to the model as prior knowledge; the second stage is the generative event relationship extraction stage. In order to match the previous task, that is, pre-training and knowledge base fine-tuning, the event relationship extraction task is converted into a generative task. Considering the efficiency and generalization of the overall training, classification is added as an auxiliary task to improve the representation on the encoder. In addition, the analysis of events in generative tasks is more closely related to the semantics of the context. In order for the model to learn more detailed event relationships, it is also necessary to adjust the event positions and supplement the relationships between them as a way of data enhancement, that is, inverting the labels. This enhancement method can be used for the MATRES dataset, but the TB-Dense dataset itself has undergone label adjustment to enrich the data, so it can be used directly. This embodiment can be used for other related event relationship extraction datasets. Only two commonly used datasets are used as examples, and the subsequent classification and generation tasks are trained by calculating the joint loss to achieve better event relationship recognition.

[0027] Step S20: Input the text data into the pre-trained language model ERNIE-GEN, so that the pre-trained language model ERNIE-GEN is combined with the common sense knowledge base ATOMIC2020 through fine-tuning and the common sense knowledge related to the event is integrated into the model.

[0028] It should be noted that the step of inputting text data into the pre-trained language model ERNIE-GEN so that the pre-trained language model ERNIE-GEN is combined with the common sense knowledge base ATOMIC2020 through fine-tuning and the common sense knowledge related to the event is integrated into the model includes: inputting text data into the pre-trained language model ERNIE-GEN so that the pre-trained language model ERNIE-GEN constructs data and trains in the common sense knowledge base ATOMIC2020; integrating the factual knowledge in the trained common sense knowledge base ATOMIC2020 into the pre-trained language model ERNIE-GEN.

[0029] It can be understood that the data in the common sense knowledge base is triple data, and the model is trained through sequence-to-sequence generation so that the model can generate missing head entities, relations or tail entities.

[0030] In the specific implementation, the text data is input into the pre-trained language model ERNIE-GEN, so that the pre-trained language model ERNIE-GEN is combined with the common sense knowledge base ATOMIC2020 through fine-tuning and the common sense knowledge related to the event is integrated into the model, specifically including: The common sense knowledge in the common sense knowledge base ATOMIC2020 is represented in the form of triples as Where head represents the head entity, relation represents the relationship, and tail represents the tail entity; The triplets are trained through generative tasks, which include the following three generative tasks: The common sense knowledge in the common sense knowledge base ATOMIC2020 is integrated into the pre-trained language model ERNIE-GEN through generative tasks.

[0031] In the specific implementation, for the first stage, the model uses ERNIE-GEN large. ERNIE-GEN is a model developed by Baidu. As a pre-training-fine-tuning framework for text generation tasks, it has shown good results in text generation tasks. It has released Chinese and English versions, which are suitable for text generation tasks in Chinese and English. For the first time, the model adds span-by-span generation tasks in the pre-training stage, allowing the model to generate a semantically complete fragment each time, which meets our task requirements. For the event relationship analysis task in English, the common sense knowledge base uses ATOMIC2020, a large common sense knowledge graph developed by AllenAI, which contains more than 1.3 million daily reasoning knowledge tuples about entities and events, and common sense knowledge graphs of daily reasoning knowledge tuples about entities and events. This knowledge base is designed to supplement the common sense knowledge encoded in the current language model, and it contains 23 different types of common sense relations, which can be broadly classified into three categories: social interaction, physical entity, and event-centric common sense, representing a large-scale text description common sense repository, which encodes the social and physical aspects of human daily experience, and can be roughly divided into three types: 9 common sense relations of social interaction, 7 common sense relations of physical entities, and 7 event-centric common sense relations around the context of a given event of interest. Among them, event-related common sense knowledge is closely related to event relationship analysis. Considering the completeness of the knowledge graph and the full training of knowledge, this stage uses the complete knowledge base data for training. For the model training of the Chinese version of the common sense knowledge base, because there is currently a lack of similar Chinese common sense knowledge bases, ATOMIC2020 can be translated. Because its content is relatively simple, there will not be much difference in the translation effect. It can be called as a Chinese common sense knowledge base. Taking the English version of the knowledge graph as an example, the data in the common sense knowledge graph is mainly in the form of triples, expressed as: , it is easier to combine with generative tasks. In the fine-tuning task, the following method is used for training. The head, tail and relationship positions are removed separately, that is, two elements are predicted to generate the third one, and a sequence-to-sequence generative task is constructed, which can be expressed as follows: The entire fine-tuning process can be expressed as follows: The data is constructed as: in is the source text, Representing head, relation and tail respectively, Represents the target text. When constructing the input, For example, the input text is: .

[0032] in All represent the corresponding input prefix, and the following and Represents the corresponding text sequence, the target text can be expressed as: ,in Represents the prefix of the output text. This part needs to be given as a signal to generate subsequent text. Indicates the corresponding tail text that needs to be generated.

[0033] The training process is: First use Encode and obtain sequence representation: Among them, X and Y are the Token serialization representations of the source text Source and the target text Target respectively. Refers to the use of Encode, and then input the encoded source text and target text's Token representation into the encoder and decoder parts of the model as the source sequence and target sequence: Among them, the embedding vector corresponding to the encoder is expressed as , represents the embedding vector of the i-th Token, n is the number of Tokens in the source text, and the embedding vector corresponding to the decoder is expressed as , Represents the embedding vector of the jth Token in the target sequence, and m is the number of Tokens in the target text.

[0034] in, represents the hidden state of the encoder at each time step.

[0035] The decoder receives the encoder’s hidden state and the embedding vector of the decoder input , calculate the decoder's hidden state and output probability distribution: in, is the hidden state of the decoder at time step t-1, and is the weight matrix and bias vector of the output layer, is the output probability distribution of time step t, generating the token of time step t according to the input source sequence and the representation of the t-1 step before time step t, and the loss is calculated using the following loss: in, is the length of the target sequence.

[0036] In the fine-tuning stage, the ATOMIC2020 common sense knowledge base is used for training, and the factual knowledge in the knowledge base is integrated into the ERNIE-GEN model. The data training of the Chinese version is similar to that of the English version. The Chinese version of the knowledge base can be obtained by translation, and then the ERNIE-GEN model on the Chinese version is fine-tuned to obtain a model enhanced with the common sense knowledge base to perform subsequent Chinese event relationship extraction tasks.

[0037] Step S30: Based on the pre-trained language model ERNIE-GEN, the event relationship extraction task is converted into a generative task to generate a temporal relationship between at least two events.

[0038] In the specific implementation, based on the pre-trained language model ERNIE-GEN, the event relationship extraction task is converted into a generative task, and the steps of generating the temporal relationship between at least two events include: adjusting the data set, based on the text Text to be analyzed in the existing data set, two events Event1 and Event2 to be analyzed, label data Label and prompt text to construct the source sequence and target sequence as follows: use Encode and obtain sequence representation: Among them, X and Y are the Token serialization representations of the source text Source and the target text Target respectively; The Token representation of the encoded source text and target text is input into the encoder and decoder parts of the model as the source sequence and target sequence: Among them, the embedding vector corresponding to the encoder is expressed as , represents the embedding vector of the i-th Token, n is the number of Tokens in the source text, and the embedding vector corresponding to the decoder is expressed as , represents the embedding vector of the jth Token in the target sequence, and m is the number of Tokens in the target text; in, represents the hidden state of the encoder at each time step; The decoder accepts the encoder's hidden state and the embedding vector of the decoder input , calculate the decoder's hidden state and output probability distribution: in, is the hidden state of the decoder at time step t-1, and is the weight matrix and bias vector of the output layer, is the output probability distribution of time step t, generating the token of time step t according to the input source sequence and the representation of the t-1 step before time step t, and the loss is calculated using the following loss: in, is the length of the target sequence.

[0039] In a specific implementation, the event pairs Event1 and Event2 in the text data to be analyzed are combined with the prompt text to construct a source sequence and a target sequence; Generate a temporal relationship label between event pairs through a generative task, where the temporal relationship label includes at least one of a sequence of event occurrence, synchronous occurrence, inclusion relationship, or uncertain relationship; The event pairs are reversed through data augmentation to generate relative time orders corresponding to different event sequences to enrich the training data.

[0040] Step S40: In the generative task, the encoder parameters of the pre-trained language model ERNIE-GEN are optimized through the auxiliary classification task.

[0041] It can be understood that a multi-layer perceptron MLP is constructed as a classifier on the last hidden state output by the encoder, and the classifier consists of a fully connected layer and a softmax layer; the contextual representation of the event pair is classified by the classifier to generate the temporal relationship category probability distribution of the event pair; the difference between the predicted category distribution and the true category label is calculated through the cross entropy loss to optimize the encoder parameters.

[0042] In a specific implementation, in a generative task, the steps of optimizing the encoder parameters of the pre-trained language model ERNIE-GEN through an auxiliary classification task include: The classification task is used as an auxiliary task and jointly trained with text generation to obtain the encoded text representation: in, represents the hidden state of the encoder at each time step; Use the last hidden state output by the encoder as the context representation of the entire input sequence: A multi-layer perceptron (MLP) is constructed on the encoder output as a classifier. The classifier consists of a fully connected layer and a softmax layer. The fully connected layer represents the context. Mapping to category scores : in, and Represent the weight matrix and bias vector of the classifier respectively; The Softmax layer converts the category scores Convert to category probability distribution p: This task uses cross entropy loss to calculate the predicted category distribution p and the true category label The difference between: in, is the number of categories, is the one-hot representation of the true category label, is the probability of the predicted category; The overall training loss during the training phase can be expressed as: in, and As a hyperparameter, it needs to be adjusted based on the training effect.

[0043] Step S50: outputting the time relationship between at least two events, where the time relationship includes at least one of a sequence of occurrence of the events, synchronous occurrence, inclusion relationship or uncertain relationship.

[0044] In the specific implementation, in the inference stage, this embodiment uses part of the prompt content as the input content of the inference stage. The model needs to continue to infer the output content based on the partial input. The goal is to infer the label part, that is, to obtain the corresponding label content. The partial prompt content here is: or , the input in the reasoning phase is the source text and part of the target text, and then it also needs to go through Encoding for serialization: in, express deal with, and They represent the source text and part of the target text that need to be input at this stage, and the token sequence is obtained after processing. and , it needs to be spliced ​​and then encoded, spliced ​​in the following form and then processed by the encoder encoding: in, Represents the sequence of input encoder, that is, after The processed text sequence, Indicates the start symbol of the sequence. Indicates the end symbol. The model can realize reasoning generation by combining the subsequent partial prompt text. The inference phase uses beam search to achieve generation. The probability of a sequence token containing the initial state and the start of decoding: in, and The tokens and corresponding probabilities of the decoded partial target text are all 1 given the sequence. Then, for each time step t of the decoder, In the bundle , calculate the probability distribution of the next Token: Expand each candidate sequence to generate new candidate sequences and their probabilities: Keep the highest probability candidate sequences, where Generally set to 3, update bundle : The decoding termination condition is when all candidate sequences generate the termination symbol Stop decoding after that or when the maximum length is reached. The maximum length is not set here, and the sequence generation is terminated only according to the termination symbol.

[0045] Then select the optimal sequence and Select the complete sequence with the highest probability as the final output: Generate the result and generate the token sequence Decoded as text: Finally, the decoded generated text is obtained as the subsequent generated text, which can be used as the result of event relationship analysis, that is, label word generation.

[0046] It should be noted that in the reasoning stage, part of the prompt content is used as input, and the temporal relationship labels of event pairs are generated through beam search; the target sequence is generated through the decoder, and the complete sequence with the highest probability is selected as the final output; the generated Token sequence is decoded into text, and the temporal relationship labels between event pairs are output.

[0047] It should be noted that in the Chinese event relationship extraction task, the common sense knowledge base ATOMIC2020 is translated to generate a Chinese version of the common sense knowledge base; the pre-trained language model ERNIE-GEN is fine-tuned through the Chinese version of the common sense knowledge base to enhance the model's common sense knowledge reserve in the Chinese event relationship extraction task.

[0048] This embodiment obtains text data to be analyzed, and inputs the text data into the pre-trained language model ERNIE-GEN, so that the pre-trained language model ERNIE-GEN is combined with the common sense knowledge base ATOMIC2020 through fine-tuning and the common sense knowledge related to the event is integrated into the model; based on the pre-trained language model ERNIE-GEN, the event relationship extraction task is converted into a generative task to generate a time relationship between at least two events; In the generative task, the encoder parameters of the pre-trained language model ERNIE-GEN are optimized through auxiliary classification tasks; the temporal relationship between at least two events is output. By combining the pre-trained language model, knowledge enhancement, prompt learning and multi-task learning, as well as the common sense knowledge base and generative tasks, better results are achieved in various application scenarios such as event relationship extraction and time series knowledge graph construction.

[0049] It is understandable that the technology of extracting temporal relations between events is committed to analyzing the temporal relations between specified events from the analysis text, that is, the relationship between the time when an event occurs or lasts and other events, which may be the time of occurrence, or simultaneous occurrence, or the inclusion relationship between durations. Of course, it may be difficult to distinguish the relationship between events due to the limited understanding of the context, which is defined as an uncertain temporal relationship. The extraction of relations between events can help understand the dynamics of complex events in natural language, which is beneficial to various downstream tasks, including the construction of event graphs, the analysis and construction of time-series knowledge graphs, the prediction of future events, as well as question-answering and summarization, information retrieval tasks, etc. Combined with existing research, the challenges faced by this technology include how to introduce common sense knowledge to enhance the model's understanding of event context under limited and diverse language expressions and improve event representation; and how to adjust the task implementation method of event relationship extraction tasks to make them more closely integrated with pre-trained language models and knowledge bases to improve the ability to handle complex temporal order relationships.

[0050] To address these challenges, this embodiment proposes an innovative generative event relation extraction technology that focuses on exogenous knowledge enhancement. This method is based on a pre-trained language model and is combined with the latest common sense knowledge base ATOMIC2020 to effectively integrate rich exogenous common sense knowledge. By carefully fine-tuning on the advanced pre-trained language model ERNIE-GEN, this technology embeds the rich information of the knowledge base into the model in the form of a generation task, enabling the model to optimize the understanding of event relations using common sense knowledge in a limited text context. In order to match the pre-training task and the method of fine-tuning the knowledge base, and to reduce the difference between the event relation extraction task and the previous task, this technology innovatively converts the traditional event relation classification task into a generation task. On the basis of the pre-trained language model, we introduced an auxiliary classification task to optimize the encoder parameters, thereby providing significant performance gains for the main task of generative event relation extraction.

[0051] This embodiment not only combines advanced natural language processing technology, but also takes into special consideration common sense knowledge that does not often appear in text contexts. By fine-tuning, these common sense knowledge bases are integrated into event relationship analysis, effectively addressing the complexity and variability of language expressions and the limited length of event context content. In addition, by converting the event relationship classification task into a generative task and combining it with the preceding task, this technology significantly reduces the gap between downstream tasks and preceding tasks, thereby greatly improving the model's effectiveness in identifying complex relationships between events. This innovative approach provides new possibilities for technological advances in event relationship extraction in the field of natural language processing.

[0052] In addition, an embodiment of the present application also proposes a computer-readable storage medium, on which is stored a program for generative event relationship extraction based on exogenous knowledge enhancement. When the program for generative event relationship extraction based on exogenous knowledge enhancement is executed by a processor, the steps of the method for generative event relationship extraction based on exogenous knowledge enhancement as described above are implemented.

[0053] Reference Figure 4 , Figure 4 This is a structural block diagram of the first embodiment of the generative event relationship extraction system based on exogenous knowledge enhancement in this application.

[0054] like Figure 4 As shown, the generative event relationship extraction system based on exogenous knowledge enhancement proposed in the embodiment of the present application includes: A data acquisition module 10 is used to acquire text data to be analyzed, wherein the text data contains at least two events; A fine-tuning module 20 is used to input text data into the pre-trained language model ERNIE-GEN, so that the pre-trained language model ERNIE-GEN is combined with the common sense knowledge base ATOMIC2020 through fine-tuning and the common sense knowledge related to the event is integrated into the model; A generation module 30, configured to convert the event relationship extraction task into a generation task based on the pre-trained language model ERNIE-GEN, and generate a temporal relationship between at least two events; An auxiliary module 40, used for optimizing encoder parameters of the pre-trained language model ERNIE-GEN through an auxiliary classification task in a generative task; The output module 50 is used to output the time relationship between at least two events, where the time relationship includes at least one of the sequence of events, synchronous occurrence, inclusion relationship or uncertain relationship.

[0055] It should be understood that the above is only an example and does not constitute any limitation on the technical solution of the present application. In specific applications, technicians in this field can make settings as needed, and the present application does not impose any limitation on this.

[0056] This embodiment obtains the text data to be analyzed, and inputs the text data into the pre-trained language model ERNIE-GEN, so that the pre-trained language model ERNIE-GEN is combined with the common sense knowledge base ATOMIC2020 through fine-tuning and the common sense knowledge related to the event is integrated into the model; based on the pre-trained language model ERNIE-GEN, the event relationship extraction task is converted into a generative task to generate the time relationship between at least two events; in the generative task, the encoder parameters of the pre-trained language model ERNIE-GEN are optimized through auxiliary classification tasks; and the time relationship between at least two events is output. By combining pre-trained language models, knowledge enhancement, prompt learning and multi-task learning, as well as common sense knowledge bases and generative tasks, better results are achieved in various application scenarios such as event relationship extraction and time series knowledge graph construction.

[0057] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present application. In practical applications, technicians in this field can select part or all of it according to actual needs to achieve the purpose of the present embodiment, and no limitation is made here.

[0058] In addition, for technical details not described in detail in this embodiment, please refer to the method for generative event relationship extraction based on exogenous knowledge enhancement provided in any embodiment of the present application, which will not be repeated here.

[0059] In addition, it should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0060] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0061] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory (ROM) / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0062] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A generative event relationship extraction method based on exogenous knowledge enhancement, characterized in that: include: Acquire text data to be analyzed, wherein the text data contains at least two events; Input the text data into the pre-trained language model ERNIE-GEN, so that the pre-trained language model ERNIE-GEN is combined with the common sense knowledge base ATOMIC2020 through fine-tuning and the common sense knowledge related to the event is integrated into the model; Based on the pre-trained language model ERNIE-GEN, converting the event relationship extraction task into a generative task to generate the temporal relationship between the at least two events; In the generative task, the encoder parameters of the pre-trained language model ERNIE-GEN are optimized through the auxiliary classification task; The time relationship between the at least two events is output, wherein the time relationship includes at least one of a sequence of occurrence of the events, synchronous occurrence, inclusion relationship, or uncertain relationship.

2. The generative event relationship extraction method based on exogenous knowledge enhancement according to claim 1 is characterized in that: The step of inputting the text data into the pre-trained language model ERNIE-GEN so that the pre-trained language model ERNIE-GEN is combined with the common sense knowledge base ATOMIC2020 by fine-tuning and integrating the common sense knowledge related to the event into the model includes: Input the text data into the pre-trained language model ERNIE-GEN so that the pre-trained language model ERNIE-GEN constructs data in the common sense knowledge base ATOMIC2020 and performs training; Incorporate factual knowledge from the trained common sense knowledge base ATOMIC2020 into the pre-trained language model ERNIE-GEN.

3. The generative event relationship extraction method based on exogenous knowledge enhancement according to claim 2 is characterized in that: The data in the common sense knowledge base is triple data, and the model is trained by a sequence-to-sequence generation method so that the model can generate missing head entities, relations or tail entities.

4. The generative event relationship extraction method based on exogenous knowledge enhancement according to claim 1 is characterized in that: Based on the pre-trained language model ERNIE-GEN, the event relationship extraction task is converted into a generative task, and the step of generating the temporal relationship between the at least two events includes: The data set is adjusted, and the source sequence and target sequence are constructed based on the text Text to be analyzed in the existing data set, the two events Event1 and Event2 to be analyzed, the label data Label and the prompt text: use Encode and obtain sequence representation: Among them, X and Y are the Token serialization representations of the source text Source and the target text Target respectively; The Token representation of the encoded source text and target text is input into the encoder and decoder parts of the model as the source sequence and target sequence: Among them, the embedding vector corresponding to the encoder is expressed as , represents the embedding vector of the i-th Token, n is the number of Tokens in the source text, and the embedding vector corresponding to the decoder is expressed as , represents the embedding vector of the jth Token in the target sequence, and m is the number of Tokens in the target text; in, represents the hidden state of the encoder at each time step; The decoder receives the encoder’s hidden state and the embedding vector of the decoder input , calculate the decoder's hidden state and output probability distribution: in, is the hidden state of the decoder at time step t-1, and is the weight matrix and bias vector of the output layer, is the output probability distribution of time step t, generating the token of time step t according to the input source sequence and the representation of the t-1 step before time step t, and the loss is calculated using the following loss: in, is the length of the target sequence.

5. The generative event relationship extraction method based on exogenous knowledge enhancement according to claim 4 is characterized in that: The step of optimizing the encoder parameters of the pre-trained language model ERNIE-GEN through the auxiliary classification task in the generative task includes: The classification task is used as an auxiliary task and jointly trained with text generation to obtain the encoded text representation: in, represents the hidden state of the encoder at each time step; Use the last hidden state output by the encoder as the context representation of the entire input sequence: A multi-layer perceptron (MLP) is constructed on the encoder output as a classifier. The classifier consists of a fully connected layer and a softmax layer. The fully connected layer represents the context. Mapping to category scores : in, and Represent the weight matrix and bias vector of the classifier respectively; The Softmax layer converts the category scores Convert to category probability distribution p: This task uses cross entropy loss to calculate the predicted category distribution p and the true category label The difference between: in, is the number of categories, is the one-hot representation of the true category label, is the probability of the predicted category; The overall training loss during the training phase can be expressed as: in, and As a hyperparameter, it needs to be adjusted based on the training effect.

6. A generative event relationship extraction system based on exogenous knowledge enhancement, characterized in that: include: A data acquisition module, used to acquire text data to be analyzed, wherein the text data contains at least two events; A fine-tuning module, used for inputting the text data into the pre-trained language model ERNIE-GEN, so that the pre-trained language model ERNIE-GEN is combined with the common sense knowledge base ATOMIC2020 through fine-tuning and the common sense knowledge related to the event is integrated into the model; A generation module, configured to convert the event relationship extraction task into a generation task based on the pre-trained language model ERNIE-GEN, and generate a temporal relationship between the at least two events; An auxiliary module, used for optimizing the encoder parameters of the pre-trained language model ERNIE-GEN through an auxiliary classification task in a generative task; The output module is used to output the time relationship between the at least two events, where the time relationship includes at least one of the sequence of event occurrence, synchronous occurrence, inclusion relationship or uncertain relationship.

7. The generative event relationship extraction system based on exogenous knowledge enhancement according to claim 6 is characterized in that: The fine-tuning module is further used to input the text data into the pre-trained language model ERNIE-GEN so that the pre-trained language model ERNIE-GEN constructs data in the common sense knowledge base ATOMIC2020 and performs training; Incorporate factual knowledge from the trained common sense knowledge base ATOMIC2020 into the pre-trained language model ERNIE-GEN.

8. The generative event relationship extraction system based on exogenous knowledge enhancement according to claim 6 is characterized in that: The generation module is also used to adjust the data set, and construct the source sequence and the target sequence based on the text Text to be analyzed in the existing data set, the two events Event1 and Event2 to be analyzed, the label data Label and the prompt text as follows: use Encode and obtain sequence representation: Among them, X and Y are the Token serialization representations of the source text Source and the target text Target respectively; The Token representation of the encoded source text and target text is input into the encoder and decoder parts of the model as the source sequence and target sequence: Among them, the embedding vector corresponding to the encoder is expressed as , represents the embedding vector of the i-th Token, n is the number of Tokens in the source text, and the embedding vector corresponding to the decoder is expressed as , represents the embedding vector of the jth Token in the target sequence, and m is the number of Tokens in the target text; in, represents the hidden state of the encoder at each time step; The decoder accepts the encoder's hidden state and the embedding vector of the decoder input , calculate the decoder's hidden state and output probability distribution: in, is the hidden state of the decoder at time step t-1, and is the weight matrix and bias vector of the output layer, is the output probability distribution of time step t, generating the token of time step t according to the input source sequence and the representation of the t-1 step before time step t, and the loss is calculated using the following loss: Where T is the length of the target sequence.

9. A computer device, characterized in that: The device comprises: a memory and a processor, wherein the processor executes the method according to any one of claims 1 to 5 when running computer instructions stored in the memory.

10. A computer-readable storage medium, characterized in that: The method comprises instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 5.

Images