Causal event extraction method, device and computer equipment based on pre-training model

Abstract

The present application relates to a causal event extraction method, device and computer equipment based on a pre-trained model. The method uses a position-based auxiliary annotation method to annotate causal pairs in the original text data; the annotated data is vectorized, the obtained text sequence is extracted using the BERT pre-trained model, and then encoded using the LSTM network, and the encoded sentence representation is respectively used in the first fully connected layer and the second fully connected layer to expand the shape of the feature into the shape of the position auxiliary tag matrix; the obtained output is processed by the third fully connected layer after the corresponding activation function, and the response time pairs are linked by indexing according to the obtained cause event matrix, relationship head matrix and effect event matrix to obtain a causal event representation. This method is used to extract event pairs with causal associations on the economic field data set at the sentence level, which improves the accuracy of causal event extraction.

Description

Technical Field

[0001] The present application relates to the technical field of information extraction, and in particular to a method, apparatus and computer device for extracting causal events based on a pre-trained model. Background Art

[0002] Early event extraction usually used rule-based pattern matching methods. With the construction of large-scale corpora and the rise of statistical learning, methods based on statistical learning have emerged. Methods based on rule-based pattern matching can achieve good results in specific fields, but their processes are complex, prone to cascading errors, and are limited to specific field text formats, which is particularly evident in the complexity and diversity of text in the economic field. Methods based on statistical learning need to rely on complex feature engineering, natural language processing tools, and a large amount of annotated data as training sets. Although there is a wealth of text data in the existing economic field, the available annotated data is relatively small, and the scale of related data sets is small. Small-scale data sets are basically unable to train complex models, resulting in unstable test results.

[0003] With the development of artificial intelligence, deep learning has made significant progress in the field of natural language processing, and event extraction methods based on deep learning have become a research hotspot and mainstream direction for event extraction. Compared with statistical learning methods, deep learning-based methods simplify feature engineering and reduce the need for domain expert knowledge. Compared with rule-based pattern matching methods and statistical learning-based methods, deep learning-based methods can more effectively capture data features through deep learning models. Summary of the invention

[0004] Based on this, it is necessary to provide a causal event extraction method, device and computer equipment based on a pre-trained model to address the above technical problems.

[0005] A causal event extraction method based on a pre-trained model, the method comprising:

[0006] The original text data to be processed is obtained, and the cause-effect pairs are marked in the original text data by using a position-based auxiliary marking method.

[0007] The labeled data is represented as a vector to obtain a text sequence.

[0008] The text sequence is used to extract text semantic features using the BERT pre-trained model to obtain word representation vectors.

[0009] The word representation vector is encoded using the LSTM network to obtain the encoded sentence representation.

[0010] The encoded sentence representation uses the first fully connected layer and the second fully connected layer respectively, and the shape of the feature is expanded into the shape of the position-assisted tag matrix.

[0011] The outputs of the two fully connected layers are processed by the corresponding activation function and then processed by the third fully connected layer to obtain the cause event matrix, the relationship head matrix and the effect event matrix.

[0012] According to the cause event matrix, relation head matrix and effect event matrix, the response time pairs are linked by indexes to obtain the cause-effect event representation.

[0013] In one embodiment, the causal event representation is performed by converting the causal event triple into a causal event triple. The causal event triple is , where C is the cause event, I is the influence, and E is the result event.

[0014] In one embodiment, original text data to be processed is obtained, and cause-effect pairs are marked in the original text data using a position-based auxiliary marking method, including:

[0015] Get the raw text data to be processed.

[0016] The cause events and effect events in the original text data are stored and marked with their event head index Head and tail index Tail.

[0017] In one embodiment, the features extracted by the two fully connected layers are processed by a third fully connected layer after being processed by corresponding activation functions to obtain a cause event matrix, a relation head matrix, and a result event matrix, including:

[0018] The features extracted by the two fully connected layers are processed by the corresponding activation function and then processed by the third fully connected layer. The extracted effect events are stored in the cause event matrix in the form of The result event matrix is ​​stored in the form of The form of is stored in the relation head matrix to represent the causal relationship between event pairs; , are the head index and tail index of the event respectively. , They are the head index and tail index of the result event respectively.

[0019] In one embodiment, according to the cause event matrix, the relation header matrix and the effect event matrix, the response time pairs are linked by indexes to obtain the cause-effect event representation, including:

[0020] If the event matrix And the result event matrix Found in the causal head matrix , then the corresponding causal triplet is extracted.

[0021] In one embodiment, the activation function is an ELU activation function.

[0022] A causal event extraction device based on a pre-trained model, the device comprising:

[0023] The annotation module is used to obtain the original text data to be processed and annotate the cause-effect pairs in the original text data using a position-based auxiliary annotation method.

[0024] The text semantic feature extraction module is used to vectorize the annotated data to obtain a text sequence; the text sequence is extracted using the BERT pre-trained model to obtain a word representation vector.

[0025] The LSTM encoding module is used to encode the word representation vector using the LSTM network to obtain the encoded sentence representation.

[0026] The feature shape expansion module is used to use the first fully connected layer and the second fully connected layer to represent the encoded sentence, respectively, to expand the shape of the feature into the shape of the position auxiliary tag matrix.

[0027] The feature matrix determination module is used to process the features extracted by the two fully connected layers through the corresponding activation function and then use the third fully connected layer to process them to obtain the cause event matrix, the relationship head matrix and the effect event matrix.

[0028] The causal event extraction module is used to link the response time pairs through indexes according to the cause event matrix, the relationship head matrix and the effect event matrix to obtain the causal event representation.

[0029] In one embodiment, the causal event representation in the causal event extraction module adopts causal event triples, and the causal event triples are , where C is the cause event, I is the influence, and E is the result event.

[0030] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0031] The original text data to be processed is obtained, and the cause-effect pairs are marked in the original text data by using a position-based auxiliary marking method.

[0032] The labeled data is represented as a vector to obtain a text sequence.

[0033] The text sequence is used to extract text semantic features using the BERT pre-trained model to obtain word representation vectors.

[0034] The word representation vector is encoded using the LSTM network to obtain the encoded sentence representation.

[0035] The encoded sentence representation uses the first fully connected layer and the second fully connected layer respectively, and the shape of the feature is expanded into the shape of the position-assisted tag matrix.

[0036] The outputs of the two fully connected layers are processed by the corresponding activation function and then processed by the third fully connected layer to obtain the cause event matrix, the relationship head matrix and the effect event matrix.

[0037] According to the cause event matrix, relation head matrix and effect event matrix, the response time pairs are linked by indexes to obtain the cause-effect event representation.

[0038] The above-mentioned causal event extraction method, device and computer equipment based on the pre-training model, the method uses a position-based auxiliary annotation method to annotate the causal pairs in the original text data; the annotated data is represented by vectors to obtain a text sequence; the text sequence is extracted using the BERT pre-training model to obtain a word representation vector; the word representation vector is encoded using an LSTM network to obtain an encoded sentence representation; the encoded sentence representation is respectively used by the first fully connected layer and the second fully connected network to expand the shape of the feature into the shape of a position auxiliary tag matrix; the outputs of the two fully connected layers are processed by the third fully connected layer after being processed by the corresponding activation function to obtain a cause event matrix, a relationship head matrix and a result event matrix; according to the cause event matrix, the relationship head matrix and the result event matrix, the response time pairs are linked by indexes to obtain a causal event representation. The method is used to extract event pairs with causal associations on a sentence-level economic field data set, thereby improving the accuracy of causal event extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 Schematic diagram of a process of a causal event extraction method based on a pre-training model in one embodiment;

[0040] Figure 2 A schematic diagram of the structure of a joint extraction model for causal event pairs in one embodiment;

[0041] Figure 3 Schematic diagram of an optimized location-assisted labeling method in an embodiment, wherein (a) is a cause event matrix, (b) is a result event matrix, and (c) is a relationship head matrix;

[0042] Figure 4 is a structural block diagram of a causal event extraction device based on a pre-training model in one embodiment;

[0043] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] In one embodiment, Figure 1 As shown, a causal event extraction method based on a pre-trained model is provided, and the method includes the following steps:

[0046] Step 100: obtaining original text data to be processed, and marking cause-effect pairs in the original text data using a position-based auxiliary marking method.

[0047] Specifically, the raw text data to be processed may be not limited to economic texts. Economic texts contain a wealth of causal events, which intuitively reveal the impact of one economic event on another economic event. However, causal events in economic texts are often not explicitly expressed, but hidden in the text.

[0048] The traditional pointer network annotation method can solve the problem of overlapping event relationships, but the pointer network annotation method requires prior knowledge of the number of causal event pairs. In actual application, the number of causal event pairs in economic texts is not fixed, and the basic pointer network annotation method is not easy to annotate. In addition, the traditional position-assisted labeling method puts all events in the same matrix and divides the judgment of causal relationships into two matrices for annotation. In the prediction stage, it will greatly increase the number of event judgments (all events are judged to see whether there is a causal relationship between them and other events). Based on the above considerations, this method designs a position-based auxiliary annotation method suitable for the causal event extraction task in this paper.

[0049] Step 102: vectorize the labeled data to obtain a text sequence;

[0050] Specifically, the annotated original text data is first cleaned, including removing irrelevant characters (such as punctuation marks, special symbols, etc.), converting to a unified format (such as unifying uppercase and lowercase letters, removing extra spaces, etc.); then the original text data is divided into smaller units, such as words or subwords; special identifiers for the beginning, end, and word separators of the text are added, and the segmented subword sequences are converted into corresponding word vector representations through the word table. .

[0051] Step 104: Use the BERT pre-trained model to extract text semantic features from the text sequence to obtain a word representation vector.

[0052] Specifically, first represent the word vector Input the encoder of the bidirectional Transformer to generate a deep bidirectional language representation and obtain the first-layer sequence feature output; then input the first-layer sequence feature output into the encoder of the bidirectional Transformer again to generate a deep bidirectional language representation and obtain the second-layer sequence feature output; after repeated processing, a word representation vector with rich contextual information can be obtained.

[0053] The main idea of ​​pre-training models represented by BERT is to use a large amount of unlabeled text for training to learn general language representations, and then fine-tune them on specific tasks to achieve excellent performance. Generally, bidirectional Transformer encoders are used to generate deep bidirectional language expressions. These encoders process the input sentences multiple times, and each time the output of the previous layer is used as input. This multi-layer processing method helps the model better understand the complex structure and contextual information in natural language texts, thereby improving the performance of the model. A key feature of BERT is that it uses a bidirectional Transformer encoder to capture bidirectional dependencies in text, which enables BERT to better understand natural language texts and obtain richer contextual information. For example, in a sentence, the meaning of a word is often determined by other words before and after it. It is difficult to handle this situation using traditional unidirectional language models because a unidirectional model can only consider contextual information in one direction. BERT's bidirectional model can consider both the previous and the following contextual information at the same time to better understand the meaning of the entire sentence.

[0054] Step 106: Encode the word representation vector using an LSTM network to obtain an encoded sentence representation.

[0055] Specifically, the word representation vector is first input into the LSTM network, and the forget gate and input gate are used to determine which neurons need to be discarded or retained at the current moment; then the current neuron state is updated according to the results of the forget gate and input gate; finally, the output gate is used to determine which neuron information can be output at the current time step. The LSTM network receives the semantic representation obtained from the upper-layer BERT model, controls the weights of each gate through hyperparameters, and finally obtains the encoded sentence representation.

[0056] Step 108: Input the encoded sentence representation into the first fully connected layer and the second fully connected layer respectively, and expand the shape of the feature into the shape of the position auxiliary labeling matrix.

[0057] Specifically, the LSTM output sequence, i.e., the encoded sentence representation, is input into the first fully connected layer and the second fully connected layer model respectively, and the weight matrix and the bias matrix are combined to carry out the matrix transformation operation, so as to expand the shape of the features with different dimensions into the shape of the position auxiliary labeling matrix with uniform dimensions. The matrix transformation formula is:

[0058] ;

[0059] in: is the output result after linear transformation of the fully connected layer, is the LSTM output sequence, is the weight matrix, is the deviation matrix.

[0060] Since the sequence dimension of the LSTM layer output is inconsistent with the dimension of the last triple matrix, but during the model training process, the sequence dimension of each component output must be consistent with the input dimension of the next component, otherwise the model cannot read the data. Therefore, a fully connected layer is added to the model, and the weight matrix is ​​multiplied by the sequence output by the previous component. In this way, the linear transformation of the matrix is ​​achieved, and the input sequence is mapped from the original feature space to the feature space of the corresponding dimension of the position auxiliary labeling matrix. The fully connected layer can not only achieve the purpose of changing the shape of the sequence output by the LSTM layer, but also reduce the computational complexity of the model and improve the generalization of the model.

[0061] Step 110: The outputs of the two fully connected layers are processed by the third fully connected layer after being processed by the corresponding activation function to obtain a cause event matrix, a relation head matrix and a result event matrix.

[0062] Specifically, three matrices are set up for each sentence, namely the event matrix, the relationship head matrix and the relationship tail matrix.

[0063] The causal event pair joint extraction model includes: BERT pre-training model, LSTM network, first fully connected layer, second fully connected layer and third fully connected layer; the structure of the causal event pair joint extraction model is as follows Figure 2 As shown in the figure. The design of the joint extraction model of causal event pairs relies on the encoding feature representation of the BERT pre-trained model and the ability of the LSTM network to solve the long-term dependency problem that is common in general neural networks. First, the data in the dataset is input into the BERT pre-trained model to obtain the semantic representation of the text sequence. Then the obtained word representation vector is input into the LSTM model to further integrate the text semantic information, and finally the result is obtained through the fully connected layer and the corresponding activation function.

[0064] Sequence the text , after the BERT pre-training model, the text semantic features can be extracted to obtain ,in Indicates The context encoding representation of a word, Indicates the maximum length of the sequence, represents the input of LSTM. Finally, the sentence encoded by LSTM can be represented as ,in represents the hidden state at the i-th time step in the sentence.

[0065] Step 112: According to the cause event matrix, the relation head matrix and the effect event matrix, the response time pairs are linked together by indexes to obtain a cause-effect event representation.

[0066] In the above-mentioned causal event extraction method based on pre-training model, the method uses a position-based auxiliary annotation method to annotate causal pairs in the original text data; the annotated data is represented by vectors to obtain a text sequence; the text sequence is extracted using the BERT pre-training model to obtain a word representation vector; the word representation vector is encoded using an LSTM network to obtain an encoded sentence representation; the encoded sentence representation is respectively extended to the shape of the feature into the shape of a position auxiliary tag matrix using the first fully connected layer and the second fully connected network; the outputs of the two fully connected layers are processed by the third fully connected layer after being processed by the corresponding activation function to obtain a cause event matrix, a relationship head matrix and a result event matrix; according to the cause event matrix, the relationship head matrix and the result event matrix, the response time pairs are linked by indexes to obtain a causal event representation. This method is used to extract event pairs with causal associations on a sentence-level economic field data set, which improves the accuracy of causal event extraction.

[0067] In one embodiment, the causal event representation in step 112 is performed by converting the causal event triple into a causal event triple. The causal event triple is , where C is the cause event, I is the influence, and E is the result event.

[0068] In one embodiment, step 100 includes: obtaining original text data to be processed; and storing and marking the cause event and the effect event in the original text data with the head index Head and the tail index Tail of the event.

[0069] In one embodiment, step 110 includes: processing the features extracted by the two fully connected layers with the corresponding activation function and then processing them with the third fully connected layer, The extracted effect events are stored in the cause event matrix in the form of The result event matrix is ​​stored in the form of The form of is stored in the relation head matrix to represent the causal relationship between event pairs; , are the head index and tail index of the event respectively. , They are the head index and tail index of the result event respectively.

[0070] In one embodiment, step 112 includes: if the event matrix And the result event matrix Found in the causal head matrix , then the corresponding causal triplet is extracted.

[0071] The optimized position-assisted marking method is as follows Figure 3 As shown, (a) is the cause event matrix, (b) is the effect event matrix, and (c) is the relationship head matrix; the cause and effect event pairs are linked through indexes to establish a causal relationship.

[0072] In one embodiment, the activation function is an ELU activation function.

[0073] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0074] In a verification example, an economic text dataset is used in this example, which contains 2,500 annotated data. The data mainly comes from financial research reports and financial news texts. The data sample is as follows:

[0075] {"id": 82;

[0076] "text": "Supply chain disruptions lead to production delays for a certain product";

[0077] "relations": [{"id": 220, "from_id": 327, "to_id": 328, "type": "Influence"}],

[0078] "entities": [{"id": 327, "start_offset": 0, "end_offset": 8, "label":"event"},

[0079] {"id": 328, "start_offset": 9, "end_offset": 17, "label": "event"}]};

[0080] Where "relations" is a set of relations, as follows:

[0081] id: relation id;

[0082] from_id: due to event id;

[0083] to_id: result event id;

[0084] type: relationship type, only "Influence" class;

[0085] "entities" is a collection of events, as follows:

[0086] id: causal event id, corresponding to from_id and to_id in the relationship set;

[0087] start_offset: the starting index of the event in the text;

[0088] end_offset: the ending index of the event in the text;

[0089] label: event type, only "event" class.

[0090] (1) Evaluation indicators

[0091] The evaluation indicators are precision (P), recall (R) and The specific calculation method is as follows:

[0092] (1)

[0093] (2)

[0094] (3)

[0095] Among them, TP is the number of extracted events predicted correctly, FP is the number of extracted events predicted incorrectly, and FN is the number of events that were not predicted but existed in the original text.

[0096] In this causal event extraction model, the following three Value as evaluation index:

[0097] 1) F1 value of causal event extraction;

[0098] 2) F1 value of causal triple extraction;

[0099] 3) Relaxed F1 value for causal triplets.

[0100] The calculation of the F1 value of causal event extraction is basically the same as the F1 value of the normal event extraction model. TP, FP, and FN are the number of all correctly predicted extracted events, the number of all incorrectly predicted extracted events, and the number of all unpredicted events that exist in the original text.

[0101] Causal triple extraction The value is also calculated similarly. TP is the number of causal event triples predicted by the model that are consistent with the actual causal event triples; FP is the number of causal event triples predicted by the model that are inconsistent with the actual causal event triples; FN is the number of causal event triples that actually exist but are missing from the predicted causal event triples.

[0102] Considering that some predicted triples differ from the actual triples by only a few words, the precise calculation method is too strict, so a relaxed version of the F1 value is used as the evaluation indicator, and the case where there is only a difference of a few words is also considered to be a correct prediction. The specific judgment basis is: when the longest common subsequence coefficient of the causal event pair in the predicted triple and the causal event pair in the standard are both greater than 0.9, the triple is considered to be correctly predicted.

[0103] These three evaluation indicators have their own advantages and disadvantages, and this experiment finally uses the average value of these three indicators. as the final evaluation criteria.

[0104] (4)

[0105] in Extract F1 value for causal events, Extract the F1 value for the causal triples, It is the relaxed version of F1 value for causal triples.

[0106] (2) Loss Function

[0107] This embodiment uses a binary cross entropy loss function + a sigmoid activation function as the loss function for loss calculation. The loss function measures the difference between the model prediction value and the true label. Its calculation process first converts the input predicted logits (logical value) into a probability through the sigmoid function, and then calculates the binary cross entropy loss based on the true label. Combining the two operations together can improve numerical stability and is more competitive than a pure binary cross entropy function.

[0108] (3) Experimental process

[0109] There are 2500 data used in this example, of which 2000 are used as training sets, 200 as validation sets, and 300 as test sets. The data exists in text format and is transferred to the model for training in batch size after data format conversion.

[0110] During the training of the model, AdamW is used as the optimizer. AdamW is a variant of the Adam optimizer, which is designed to solve the problems that may occur during the training of the Adam optimizer, such as instability and decreased convergence speed. The AdamW optimizer uses a new method in weight decay to maintain the regularization effect of the weights, and has better stability and faster convergence speed than the Adam optimizer. This optimizer setting can improve the training speed and effect of the model, especially when dealing with large-scale data sets and deep neural networks.

[0111] Define the weights of bias and LayerNorm as parameter types that do not require weight decay, and then put the parameters that require weight decay and those that do not require weight decay into two dictionaries respectively, where the relevant weight decay parameters are set for the parameters that require weight decay, and the weight decay parameters for the parameters that do not require weight decay are set to 0. Finally, use the AdamW optimizer, pass the above two dictionaries as parameters, and then specify the learning rate, weight decay and other related parameters for model training.

[0112] When training a neural network, the gradient descent method is used to update the parameters of the network to minimize the loss function. This embodiment uses the clip_grad_norm_ function to control the gradient norm to clip the gradient.

[0113] This embodiment uses ReduceLROnPlateau learning rate scheduler to dynamically adjust the learning rate of the optimizer. The main function of ReduceLROnPlateau learning rate scheduler is to monitor the indicators on the validation set, and reduce the learning rate when no better indicators are obtained for several consecutive rounds.

[0114] (4) Experimental results

[0115] For a simple causal event sentence "Lower interest rates lead to lower loan costs for home purchases.", the model predicts the output: "pred_list": [["Lower interest rates", "Influence", "Lower loan costs for home purchases"]], extracting the causal event triples and completing the expected task.

[0116] For the sentence with multiple causes and one effect, "The decline in corporate profits is the result of multiple factors, including rising raw material costs, tight labor market, and supply chain disruptions.", the model predicts the output: "pred_list": [["Raw material costs rise", "Influence", "Corporate profits fall"], ["Labor market tight", "Influence", "Corporate profits fall"], ["Supply chain disruptions", "Influence", "Corporate profits fall"]], extracting the causal event triples and completing the expected task.

[0117] For the sentence "In view of the growth of market demand and the increase of enterprise production capacity, this strategic adjustment has strengthened its competitive position in the market.", in which the event is both a cause and an effect, the model predicts the output as follows: "pred_list":[["Growth of market demand", "Influence", "Increase of enterprise production capacity"], ["Increase of enterprise production capacity", "Influence", "Strengthened its competitive position in the market"]], and extracts the causal event triples, completing the expected task.

[0118] In one embodiment, Figure 4 As shown, a causal event extraction device based on a pre-training model is provided, comprising: a labeling module, a text semantic feature extraction module, an LSTM encoding module, a feature shape expansion module, a feature matrix determination module and a causal event extraction module, wherein:

[0119] The annotation module is used to obtain the original text data to be processed and annotate the cause-effect pairs in the original text data using a position-based auxiliary annotation method.

[0120] The text semantic feature extraction module is used to vectorize the annotated data to obtain a text sequence; the text sequence is extracted using the BERT pre-trained model to obtain a word representation vector.

[0121] The LSTM encoding module is used to encode the word representation vector using the LSTM network to obtain the encoded sentence representation.

[0122] The feature shape expansion module is used to use the first fully connected layer and the second fully connected layer to represent the encoded sentence, respectively, to expand the shape of the feature into the shape of the position auxiliary tag matrix.

[0123] The feature matrix determination module is used to process the features extracted by the two fully connected layers through the corresponding activation function and then use the third fully connected layer to process them to obtain the cause event matrix, the relationship head matrix and the effect event matrix.

[0124] The causal event extraction module is used to link the response time pairs through indexes according to the cause event matrix, the relationship head matrix and the effect event matrix to obtain the causal event representation.

[0125] In one embodiment, the causal event representation in the causal event extraction module adopts causal event triples, and the causal event triples are , where C is the cause event, I is the influence, and E is the result event.

[0126] In one embodiment, the marking module is further used to obtain the original text data to be processed; and store and mark the cause events and effect events in the original text data with their event head index Head and tail index Tail.

[0127] In one embodiment, the feature matrix determination module is further used to process the features extracted by the two fully connected layers through the corresponding activation function and then use the third fully connected layer to process the features extracted by the two fully connected layers. The extracted effect events are stored in the cause event matrix in the form of The result event matrix is ​​stored in the form of The form of is stored in the relation head matrix to represent the causal relationship between event pairs; , are the head index and tail index of the event respectively. , They are the head index and tail index of the result event respectively.

[0128] In one embodiment, the causal event extraction module is also used if the causal event matrix And the result event matrix Found in the causal head matrix , then the corresponding causal triplet is extracted.

[0129] In one of the embodiments, the activation function in the feature matrix determination module is an ELU activation function.

[0130] For the specific limitations of the causal event extraction device based on the pre-trained model, please refer to the limitations of the causal event extraction method based on the pre-trained model above, which will not be repeated here. Each module in the above-mentioned causal event extraction device based on the pre-trained model can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0131] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a causal event extraction method based on a pre-trained model is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0132] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0133] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above method embodiment when executing the computer program.

[0134] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0135] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A causal event extraction method based on a pre-trained model, characterized in that: The method comprises: Acquire original text data to be processed, and mark cause-effect pairs in the original text data using a position-based auxiliary marking method; The labeled data is represented as a vector to obtain a text sequence; Extract text semantic features from the text sequence using the BERT pre-trained model to obtain a word representation vector; Encode the word representation vector using an LSTM network to obtain an encoded sentence representation; The encoded sentence representation is respectively applied to the first and second fully connected layers to expand the shape of the feature into the shape of the position-assisted tag matrix; The outputs of the two fully connected layers are processed by the third fully connected layer after being processed by the corresponding activation function to obtain the cause event matrix, the relation head matrix and the effect event matrix; According to the cause event matrix, the relation header matrix and the effect event matrix, the response time pairs are linked together by indexes to obtain a cause-effect event representation.

2. The causal event extraction method based on the pre-training model according to claim 1 is characterized in that: The causal event is represented by a causal event triple, which is: , where C is the cause event, I is the influence, and E is the result event.

3. The causal event extraction method based on the pre-training model according to claim 1 is characterized in that: Obtaining raw text data to be processed, and marking causal pairs in the raw text data using a position-based auxiliary marking method, including: Get the raw text data to be processed; The cause event and the effect event in the original text data are stored and marked with their event head index Head and tail index Tail.

4. The causal event extraction method based on the pre-training model according to claim 3 is characterized in that: The features extracted by the two fully connected layers are processed by the third fully connected layer after being processed by the corresponding activation function to obtain the cause event matrix, the relationship head matrix and the effect event matrix, including: The features extracted by the two fully connected layers are processed by the corresponding activation function and then processed by the third fully connected layer. The extracted effect events are stored in the cause event matrix in the form of The result event matrix is ​​stored in the form of The form of is stored in the relation head matrix to represent the causal relationship between event pairs; , are the head index and tail index of the event respectively. , They are the head index and tail index of the result event respectively.

5. The causal event extraction method based on the pre-training model according to claim 4 is characterized in that: According to the cause event matrix, the relationship header matrix and the effect event matrix, the response time pairs are linked by indexes to obtain a cause-effect event representation, including: If the event matrix And the result event matrix Found in the causal head matrix , then the corresponding causal triplet is extracted.

6. The causal event extraction method based on the pre-training model according to claim 1 is characterized in that: The activation function is the ELU activation function.

7. A causal event extraction device based on a pre-trained model, characterized in that: The device comprises: A tagging module, used for acquiring raw text data to be processed, and tagging cause-effect pairs in the raw text data using a position-based auxiliary tagging method; The text semantic feature extraction module is used to vectorize the annotated data to obtain a text sequence; the text sequence is extracted with the BERT pre-trained model to obtain a word representation vector; An LSTM encoding module, used for encoding the word representation vector using an LSTM network to obtain an encoded sentence representation; A feature shape expansion module, used to use the first fully connected layer and the second fully connected layer to represent the encoded sentence, and expand the shape of the feature into the shape of the position auxiliary tag matrix; A feature matrix determination module is used to process the features extracted by the two fully connected layers through the corresponding activation function and then use the third fully connected layer to obtain a cause event matrix, a relation head matrix and a result event matrix; The causal event extraction module is used to link the response time pairs through indexes according to the cause event matrix, the relationship header matrix and the effect event matrix to obtain the causal event representation.

8. The device according to claim 7, characterized in that The causal event in the causal event extraction module is represented by converting the causal event triple into a causal event triple. The causal event triple is: , where C is the cause event, I is the influence, and E is the result event.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the causal event extraction method based on the pre-training model described in any one of claims 1 to 6 is implemented.

Images