A Convergence Acceleration Method for Complex Causal Relationship Extraction Models

Abstract

The present invention discloses a convergence acceleration method for complex causal relationship extraction models. First, positive and negative samples are constructed for data enhancement. Second, causal relationships in texts are extracted through a contrastive representation learning framework. Finally, the convergence speed of complex causal relationship extraction by contrastive representation learning is improved by introducing diversity self-paced learning. The present invention greatly improves the convergence speed of complex causal relationship extraction models based on contrastive representation learning. Through testing on multiple data sets, the extraction accuracy is as high as 90%.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a convergence acceleration method for a complex causal relationship extraction model. Background Art

[0002] In natural language processing, events are very important as a form of information representation and are receiving more and more attention. Events contain a large number of internal components (such as participants, time, place, etc.) and external associations (such as semantic relationships such as causality, co-reference, and time sequence). As an important relationship between events, causality is increasingly used in human-computer interaction, aviation safety, medicine, economy, politics, social discovery, and natural science. Extracting causal relationships from text is an important part of causal discovery. Currently, causal relationship extraction in text mainly uses supervised deep learning methods based on convolutional neural networks, recurrent neural networks, and Transformer for extraction. Slow convergence speed and convergence long tail effect are important problems encountered by existing causal relationship extraction models. Therefore, how to accelerate the convergence of complex causal relationship extraction models has become a practical problem that needs to be solved urgently.

[0003] In the prior art, the extraction of complex causal relationships is mainly divided into causal reasoning, text segmentation and Tag2Triplet methods. Xia et al. proposed to use a joint structure of a bidirectional long short-term memory network and a Transformer to realize the recognition of causal pairs in text, but it requires manual annotation of the position information of the arguments in the text, which is a large workload. Christopoulou et al. proposed a walk-based model to realize causal reasoning in text. Wang et al. first performed text segmentation and then used a pre-trained Transformer model to collect multiple pairs of causal relationships in the text. Li et al. used the Tag2Triplet method to detect complex causal relationships in text, but it only works for the causal relationship of short sentences and cannot be used for complex causal processing. There are still difficulties in using relational reasoning for large-scale causal extraction. The way of segmenting text is not conducive to the extraction of causal relationships between sentences and cross-sentence causal relationships. Combining position information requires a lot of manual annotation work, which is very difficult for large-scale complex causal extraction. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the present invention provides a convergence acceleration method for complex causal relationship extraction models. First, positive and negative samples are constructed for data enhancement. Second, the causal relationship in the text is extracted through a contrastive representation learning framework. Finally, the convergence speed of complex causal relationship extraction by contrastive representation learning is improved by introducing diverse self-paced learning. The present invention greatly improves the convergence speed of the complex causal relationship extraction model based on contrastive representation learning. Through testing on multiple data sets, the extraction accuracy is as high as 90%.

[0005] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0006] Step 1: Building a data augmentation template set;

[0007] Given a causal relationship dataset Select some examples with explicit causal relationships and extract the causal argument x c , Result Theorem x e and explicit connectives ω to construct causal triplets for data augmentation Where m represents the number of triplets in D;

[0008] Step 2: Use data enhancement methods to construct positive and negative samples for contrastive representation learning;

[0009] Given an input text Get the causal argument R in the input text sequence = {x c ,x e}, and for the causal element x c and the consequentialist x e The vector representation x c and x e Perform vector concatenation, represented by X R =Concat(x c ,x e );

[0010] For every pair of causal arguments x in D ci and the consequentialist x ei Vector representation, respectively x ci and x ei , and concatenate them into vectors to represent them as X i =Concat(x ci ,x ei ), then calculate X R and X i The cosine similarity is used to measure the semantic similarity, and finally the cosine similarity with X is selected. R X with the largest cosine similarity i The corresponding conjunction is the most appropriate conjunction ω p; Positive samples are represented by Changing the directionality of arguments to construct negative samples

[0011] Step 3: Extraction of complex causal relationships;

[0012] The original input and positive samples or the original input and negative samples Input into two deep encoders of contrastive representation learning framework for relation extraction;

[0013] The two deep encoders use a two-layer structure of BiLSTM and Transformer, using contrast loss and maximum likelihood loss To construct the objective function

[0014]

[0015]

[0016]

[0017] in Represents the input vectors of the two deep encoders and vector The Euclidean distance between them, t>0 is the defined radius threshold, if the semantic similarity between the two vectors output by the model is greater than 0.5, Y=1; otherwise Y=0; α represents the weight between the two loss functions; y i and Represent the true label and predicted label of the sample respectively, and n represents the number of samples;

[0018] Step 4: Diversity self-paced learning convergence acceleration;

[0019] Introducing diverse self-paced learning to the objective function Improved by minimizing the diversity self-paced loss function Jointly train the parameters w of the complex causal relationship extraction model and the potential weight vector v = [v1,...,v n ]:

[0020]

[0021] Among them, λ and γ represent parameters that control the difficulty of samples. represents the diversity term, b represents the number of classes during training;

[0022] Solved by iterative biconvex optimization strategy When v is fixed, As a standard supervised learning objective function, the parameter w is learned with the minimum loss; for a given w, by Calculate the global optimum in It is a door unit.

[0023] The beneficial effects of the present invention are as follows:

[0024] After introducing diversity self-paced learning, the convergence speed of the complex causal relationship extraction model based on contrastive representation learning is greatly improved. Through testing on multiple data sets, the extraction accuracy is as high as 90%. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is the overall framework diagram of the method of the present invention. DETAILED DESCRIPTION

[0026] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0027] The present invention is a convergence acceleration method for complex causal relationship extraction model, which involves natural language processing knowledge discovery, causal relationship and information extraction using deep learning methods, and deep learning model performance improvement. The specific steps of the method are: first, in order to extract complex causal relationship information in text data, a complex causal relationship extraction technology based on contrastive representation learning is designed. The causal arguments in the text are extracted, and the data enhancement technology is used to add explicit causal relationship connectors, such as Because, Cause, etc., to the implicit causal relationship in the extracted argument set, and convert it into an explicit causal relationship. Then, the original text and the data-enhanced text are input into a two-layer deep network module based on a long short-term memory neural network (Bidirectional Long-Short Term Memory, BILSTM) and Transformer. The BiLSTM layer performs preliminary feature extraction on the text at the sentence level, and each sentence corresponds to a BiLSTM module. Then, each sentence feature is input into the Transformer layer, and the semantic and grammatical features at the sentence or paragraph level are extracted to obtain the causal relationship information of the text. Finally, a method based on diversity self-paced learning is used to accelerate the deep learning module. The self-paced learning module first divides the training data into different classes, and screens the original data set through two stages of between-class sampling and within-class sampling, so as to achieve the goal of building a lightweight data set for model training and improve the convergence speed of the model. The present invention mainly aims at the problem of slow model convergence caused by large data scale and uneven data samples in the process of complex causal relationship extraction. It proposes a convergence acceleration method for complex causal relationship extraction models, which uses diversity self-paced learning to improve the convergence speed of the comparative representation learning framework based on BiLSTM and Transformer, thereby improving the efficiency and performance of complex causal relationship extraction.

[0028] The present invention is mainly aimed at the slow convergence speed and long tail problem in the convergence process when using deep learning models to extract complex causal relationships in the field of natural language processing information extraction, and improves the convergence speed of the causal relationship extraction model under complex conditions such as complex argument types and directions and complex argument interactions, and eliminates the long tail effect in the training process. It uses a contrastive representation learning framework based on BiLSTM and Transformer to extract complex causal relationships in texts, and the use of contrastive representation learning can reduce the dependence on manual annotation work during the extraction process. Diversity self-paced learning is introduced to improve the convergence speed of the contrastive representation learning model in the extraction process. Diversity self-paced learning divides all training samples into different classes, measures the complexity of samples in each class, and adds a contrastive representation learning model to learn these samples from easy to difficult, while considering inter-class and intra-class features, and completing the efficient extraction of complex causal relationships in texts. Through this method, after the introduction of diversity self-paced learning, the convergence speed of the complex causal relationship extraction model based on contrastive representation learning is greatly improved. By testing on multiple data sets, the extraction accuracy is as high as 90%.

[0029] The present invention mainly uses two technologies:

[0030] Complex causal relationship extraction technology based on contrastive representation learning: Adopting the contrastive representation learning framework, constructing a set of positive and negative samples of complex causal relationships in the text, using the Transformer-based deep learning module to learn the differences between the original text and the positive and negative sample sets to identify the complex causal relationship features contained in the text, output the text similarity between the original text and the positive and negative samples, and obtain the complex causal relationship pairs contained in the text.

[0031] Diversity self-paced learning model convergence acceleration technology: Introduce diversity self-paced learning technology into the complex causal relationship extraction technology based on contrastive representation learning, evaluate the complexity of the data set, and allow the deep learning model to learn from easy to difficult through inter-class learning and intra-class learning, shorten the model learning time, accelerate model convergence, and improve model performance.

[0032] A convergence acceleration method for complex causal relationship extraction model, comprising the following steps:

[0033] Step 1: Data augmentation template set construction. Given a causal relationship dataset Select a portion of the samples with explicit causal relationships and extract the causal argument x c , Result Theorem x e and explicit connectives ω to construct causal triplets for data augmentation where m represents the number of triplets in D.

[0034] Step 2: Use data enhancement methods to construct positive and negative samples for contrastive representation learning. Given an input text Get the causal argument R in the input text sequence = {x c ,x e}, and for the causal element x c and the consequentialist x e The vector representation x c and x e Perform vector concatenation, represented by X R =Concat(x c ,x e ). In order to select the most appropriate causal connective, for each pair of causal arguments x in D ci and the consequentialist x ei Vector representation, respectively x ci and x ei , and concatenate them into vectors to represent them as X i =Concat(x ci ,x ei ), then calculate X R and X i The cosine similarity is used to measure the semantic similarity, and finally the most appropriate conjunction ω is selected according to the maximum cosine similarity. p . The positive sample can be expressed as Changing the directionality of arguments to construct negative samples

[0035]

[0036] Step 3: Extraction of complex causal relationships. and positive samples Or negative samples The input is sent to two deep encoders in the contrastive representation learning framework for relation extraction. The two deep encoders use a two-layer structure of BiLSTM and Transformer, and use contrastive loss. and maximum likelihood loss To construct the objective function

[0037]

[0038]

[0039]

[0040] in Represents the input vectors of the two deep encoders and vector The Euclidean distance between them, t>0 is the defined radius threshold, if the semantic similarity between the two vectors output by the model is greater than 0.5, Y=1; otherwise Y=0; α represents the weight between the two loss functions; y i and Represent the true label and predicted label of the sample respectively, and n represents the number of samples;

[0041] Step 4: Diversity self-paced learning convergence acceleration;

[0042] Introducing diverse self-paced learning to the objective function Improved by minimizing the diversity self-paced loss function Jointly train the parameters w of the complex causal relationship extraction model and the potential weight vector v = [v1,...,v n ]:

[0043]

[0044] Among them, λ and γ represent parameters that control the difficulty of samples. represents the diversity term and b represents the number of classes during training. The goal of the diversity self-paced learning is to minimize the weighted training loss together with the negative l1-norm regularizer. An iterative biconvex optimization strategy is used to solve Its main idea can be described as: when v is fixed, It can be used as a standard supervised learning objective function to learn the parameter w with the minimum loss. Next, for a given w, it can be obtained by To calculate the global optimum in is a gate unit that controls whether a sample should be used in the training dataset based on its simplicity.

Claims

1. A convergence acceleration method for complex causal relationship extraction models, characterized in that: The following steps are involved: Step 1: Building a data augmentation template set; Given a causal relationship dataset Select some examples with explicit causal relationships and extract the causal argument x c , Result Argument x e and explicit connectives ω to construct causal triplets for data augmentation Where m represents the number of triplets in D; Step 2: Use data enhancement methods to construct positive and negative samples for contrastive representation learning; Given an input text χ={..., <e i >x c < / e i >,..., <e j >x e < / e j >,...}, obtain the causal argument R={x c ,x e }, and for the causal element x c and the consequentialist x e The vector representation x c and x e Perform vector concatenation, represented by X R =Concat(x c ,x e ); For every pair of causal arguments x in D ci and the consequentialist x ei Vector representation, respectively x ci and x ei , and concatenate them into vectors to represent them as X i =Concat(x ci ,x ei ), then calculate X R and X i The cosine similarity is used to measure the semantic similarity, and finally the cosine similarity with X is selected. R X with the largest cosine similarity i The corresponding conjunction is the most appropriate conjunction ω p ; Positive samples are represented by Changing the directionality of arguments to construct negative samples Step 3: Extraction of complex causal relationships; The original input and positive samples or the original input and negative samples Input into two deep encoders of contrastive representation learning framework for relation extraction; The two deep encoders use a two-layer structure of BiLSTM and Transformer, using contrast loss and maximum likelihood loss To construct the objective function in Represents the input vectors of the two deep encoders and vector The Euclidean distance between them, t>0 is the defined radius threshold, if the semantic similarity between the two vectors output by the model is greater than 0.5, Y = 1; otherwise Y = 0; α represents the weight between the two loss functions; y i and Represent the true label and predicted label of the sample respectively, and n represents the number of samples; Step 4: Diversity self-paced learning convergence acceleration; Introducing diverse self-paced learning to the objective function Improved by minimizing the diversity self-paced loss function Jointly train the parameters w of the complex causal relationship extraction model and the potential weight vector v = [v1,...,v n ]: Among them, λ and γ represent parameters that control the difficulty of samples. represents the diversity term, b represents the number of classes during training; Solved by iterative double convex optimization strategy When v is fixed, As a standard supervised learning objective function, the parameter w is learned with the minimum loss; for a given w, by Calculate the global optimum in It is a door unit.

Images