A causal relationship extraction method based on BERT semantic enhancement

Abstract

The present invention discloses a causal relationship extraction method based on BERT semantic enhancement. The causal relationship extraction method comprises: a causal relationship candidate word library, BERT pre-training, and causal relationship extraction. The method is an information extraction technology for quickly extracting causal relationships in texts. The core task is to establish a basic model and an enhanced model under the architecture of the LeakGAN adversarial neural network model to perform adversarial learning to obtain highly discriminative features, analyze the causal relationships in the comment text, and achieve deep extraction under semantic enhancement. The method is based on the adversarial learning of adversarial neural networks to obtain more discriminative features, improve the accuracy of causal relationship extraction, and can be applied to event prediction, question-answering systems, and scenario generation.

Description

Technical Field

[0001] The present invention relates to the field of causal relationship extraction, and in particular to a causal relationship extraction method based on BERT semantic enhancement. Background Art

[0002] In recent years, causal relationship extraction technology has had an impact on all aspects of natural language processing tasks and has been widely used. Due to the uniqueness and diversity of causal relationship patterns in different fields, the complexity of the semantic structure of comment texts, and the diversity of expressions, it is inevitable that the difficulty of causal relationship extraction in comment texts in different fields has increased.

[0003] Causal extraction technology has high application value in event prediction, question-answering systems, and scenario generation. At the same time, due to the large amount of redundant information in comment text, it is necessary to use machine learning to quickly extract valuable information, so causal extraction is an important task. Comment texts have the characteristics of complex proper nouns, low value density, and fragmented information, which makes the text semantically ambiguous, resulting in low accuracy of causal extraction, making causal extraction also a challenging task.

[0004] Aiming at the defect of semantic ambiguity in existing causal relationship extraction that makes feature learning difficult, a causal relationship extraction method based on BERT semantic enhancement is proposed. Summary of the invention

[0005] The purpose of the present invention is to provide a causal relationship extraction method based on BERT semantic enhancement. This is a method for addressing the defect of semantic fuzzy feature learning difficulty in existing causal relationship extraction, combining BERT pre-training technology and LeakGAN adversarial neural network model, and proposing a semantically enhanced causal relationship extraction method. The main task of this method is to use the constructed causal relationship extraction model to learn features with high discrimination, and at the same time combine the causal relationship candidate word library to improve the accuracy of causal extraction. In order to achieve the above functions, the technical solution provided by the present invention includes the following steps:

[0006] 1) Construct a causal relationship candidate word library. This paper uses a method that combines causal relationship prior knowledge with semantic data to extract causal relationship candidate words from financial field review texts. During the BERT pre-training process, the three extracted candidate words with the highest matching degree are selected and passed into BERT to better learn complex proper nouns in sentences.

[0007] 2) Construct a causal relationship extraction model. In order to enable the model to learn more semantic information, the method of repeatedly describing causal relationships is adopted to enhance the representation of text semantics. At the same time, in order to better learn features, multi-feature fusion is performed to further solve the impact of fuzzy semantics on causal relationship extraction, helping users to extract causal relationships more efficiently.

[0008] In the construction of the candidate causal relationship word library, since causal relationships are usually phrases with greater emotional polarity strength, the present invention will give priority to the emotional polarity strength of the phrase. Since causal relationships may be positive emotions or negative emotions, it is not only considered whether it is a positive emotional word or a negative emotional word, but more consideration is given to the emotional degree of the phrase. The Hownet emotional dictionary of CNKI is selected. First, the frequency of emotional words in the emotional dictionary in the financial field review corpus is calculated, and then the emotional intensity of causal relationship phrases (the Emotional Intensity of Causal Words, EIC) is calculated by combining the emotional intensity manually annotated in the comment text. 2000 words with larger values ​​are screened out to form the emotional intensity word library of causal relationship words.

[0009] Furthermore, the sentiment intensity is divided into four levels, namely level 0, level 1, level 2, and level 3. The basis for the division mainly considers the following two aspects: the weighted average of the sentiment intensity of each word in the phrase and the frequency of occurrence of the phrase in the entire annotated corpus.

[0010] Furthermore, the emotional intensity of causal words (EIC) is used to measure the emotional intensity of causal word i in the annotated corpus, that is, to calculate the emotional polarity intensity of causal words from causal words. With the help of emotional intensity features, causal words can be extracted more accurately, thereby improving the accuracy of extraction. For example, the formula EIC i =|F i *I i |*E i *W i .

[0011] Furthermore, in order to make the weight more discriminative, the emotional intensity level is mapped to the range of 1 to 4, and integers are selected at the same time, so the weights are 0.5, 1, 1.5, and 2. For each increase in emotional intensity level, the weight increases by 1. On this basis, the candidate seed word library is then clustered using the K-means clustering algorithm to screen out the final candidate word library.

[0012] The causal relationship extraction model has the following process:

[0013] 1) Data preprocessing. The main tasks of data preprocessing include two aspects: one is to conduct preliminary screening of the content of the text, delete the sentence components by default or unify the sentence format; the other is to mark the screened sentences.

[0014] 2) BERT pre-training, pre-training for both the causal enhancement model and the basic model

[0015] 3) Feature fusion based on causal relationships, in addition to basic position features, in order to select more accurately, the relevance of words is considered to measure the relevance between causal words and context words, the sentiment intensity of words and the weight of the words themselves.

[0016] 4) Feature learning for causal relationship extraction. BALG (Bi-LSTM+Attention+LeakGAN, BALG) refers to learning the semantic features of complete sentences through a Bi-LSTM bidirectional recurrent network, adding a word-level attention layer to focus the feature vector on words that play a key role in causal relationship extraction, and LeakGAN implements text adversarial learning to learn high-discrimination features.

[0017] 5) Causal relationship serialization output, using CRF for serialization annotation, annotating each word in the sentence and extracting the causes and results.

[0018] The advantages and positive effects of the present invention are: a method for BERT pre-training model is proposed. This paper combines BERT pre-training with the LeakGAN network, uses BERT pre-training to obtain word vectors containing semantics, and obtains high-discrimination features for causal relationship extraction through learning of the LeakGAN adversarial neural network, further improving the accuracy of one cause and one effect extraction. A causal relationship candidate word library is constructed. Based on the causal relationship candidate word library, it can help software platforms or relevant departments to extract causal relationships, so as to effectively manage the market and help make the best decisions. At the same time, the constructed word library can also provide a reference for researchers to study causal relationships in other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of the causal relationship extraction method based on BERT semantic enhancement;

[0020] Figure 2 This is a feature fusion diagram of the causal relationship extraction method based on BERT semantic enhancement;

[0021] Figure 3 This is an example effect diagram of the causal relationship extraction method based on BERT semantic enhancement. DETAILED DESCRIPTION

[0022] The present invention will be further described below:

[0023] Figure 1 The flowchart of the causal relationship extraction method based on BERT semantic enhancement is shown in the figure. First, the proper nouns in the field are learned through the causal relationship candidate word library, the features of the proper nouns are learned in the BERT pre-training, and the pre-trained word vectors are input into the Bi-LSTM network to extract text features. At the same time, multi-feature fusion is performed to learn more features; then, features are further extracted through the adversarial neural network; finally, CRF is used for serialization output to achieve causal relationship extraction.

[0024] Among them, the main tasks of data preprocessing of the present invention include two aspects: one is to conduct preliminary screening of the content of the text, delete the sentence components by default or unify the sentence format; the other is to mark the screened sentences. Since this article involves sequence marking, punctuation marks are also marked like words (marked as "O").

[0025] Specifically, since the proper nouns in the review text are relatively complex and diverse, in order to better learn these proper nouns, the present invention proposes to construct a causal relationship candidate word library, which can better identify proper nouns during BERT pre-training and learn high-quality word vectors.

[0026] The Emotional Intensity of Causal Words (EIC) proposed in this method is used to measure the emotional intensity of causal word i in the annotated corpus, that is, to calculate the emotional polarity intensity of causal words from causal words. With the help of the emotional intensity feature, causal words can be extracted more accurately and the extraction accuracy can be improved. For example, the formula EIC i =|F i *I i |*E i *W i As shown. Using the sentiment intensity value calculated by the formula, a causal relationship seed word library can be established, and appropriate weights can be given to phrases with different sentiment intensity levels. In order to make the weight more discriminative, the sentiment intensity level is mapped to the range of 1 to 4, and integers are selected at the same time. Therefore, the weights are 0.5, 1, 1.5, and 2. For each increase in sentiment intensity level, the weight increases by 1. On this basis, the K-means clustering algorithm is then used to perform clustering calculations on the candidate seed word library to screen out the final candidate word set.

[0027] Furthermore, for feature learning of causal relationship extraction, BALG (Bi-LSTM+Attention+LeakGAN, BALG) is used, which means learning the semantic features of the complete sentence through the Bi-LSTM bidirectional recurrent network, adding a word-level attention layer to focus the feature vector on words that play a key role in causal relationship extraction, and LeakGAN implements text adversarial learning to learn high-discrimination features.

[0028] Specifically, the adversarial neural network is mainly composed of a basic model, a causal enhancement model and a softmax classifier. When using the enhancement model, the extracted causal candidate word library needs to be filled in according to the original sentence to improve the semantics. LeakGAN generates a model of adversarial network. The Manager module in the generator receives the feature vector from the discriminator and generates the target embedding and Worker module. Different from the typical reinforcement learning (RL) setting, adversarial text generation uses D φ As a learning reward function. φ It can be decomposed into a feature extractor F and a Sigmod classification layer with a final weight vector v. Given an input, there is D φ (x) = sigmoid(v T F(x;φ f ))=sigmoid(v T f) where v is the final weighted vector, φ = (φ f ,φ v ), sigmoid(z)=1 / (1+e -z ), f=F(v;φ f ) is the feature vector of the last layer, which will be sent to the generator. The generator receives the message leaked by the discriminator and regenerates data closer to the real data. In order to better converge the function, LeakGAN proposes a cross-training scheme. Supervised training (i.e. MLE) and adversarial training (i.e. GAN) are used alternately.

[0029] Specifically, the entire adversarial neural network consists of a BERT pre-trained word vector layer, a Bi-LSTM layer, a LeakGAN layer, and a softmax output classification layer. The last Softmax classification layer is shared by two models and is also called a classifier. The BERT pre-trained word vector transforms the sentence semantics into a word sequence and inputs it into the network. It performs a mask on a word in the sentence to obtain the similarity of each word, and makes a judgment through the classifier, that is, minimizing the cross entropy loss: Loss B =minmise(S(H B (x b θB ),;θ C ),y) where x b The input of the basic model features, θ B Represents the basic model

[0030] Specifically, for the enhanced model, learning is also performed by minimizing the cross entropy loss, but the parameters are different. R =minmise(S(H R (x r θ R ),y) The BERT pre-training process in the enhanced model is to add causal relationships to the original text and then perform pre-training. The training goal of the adversarial neural network is to make the basic model and the enhanced model as close as possible to the data, so that the loss function is minimized. From the parameters, θ C represents the parameters of the classifier, and y represents the real word vector. This optimizes the model parameters and improves the model accuracy.

[0031] Figure 2 This is a feature fusion diagram of the causal relationship extraction method based on BERT semantic enhancement. Although traditional causal extraction also uses the syntactic structure and semantic information of sentences, it often only uses single position features for learning, making the learned feature dimensions insufficient. Therefore, this paper uses multi-feature fusion technology to learn features with more dimensions.

[0032] Specifically, in addition to the basic position features, in order to select more accurately, the relevance of words is considered to measure the relevance between causal words and context words. The relevance is mainly calculated by calculating the similarity, as shown in the formula As shown in the figure, Ru represents the average value of causal words in the entire review corpus, Ru,i represents the correlation between word i and the average value of causal words, and Ru,j represents the correlation between word j and the average value of causal words.

[0033] Furthermore, the feature fusion based on causality, in addition to the word position features and correlation features introduced above, also includes the word sentiment intensity and the weight of the word itself.

[0034] Figure 3This is an example effect diagram of the causal relationship extraction method based on BERT semantic enhancement. Since the comment text in the financial field is selected, the data sets FNP 2021 Shared Task 2 and FNP 2020 Shared Task 2 are used. The parameter optimization route of the method disclosed in the present invention on the example data set is more efficient. In the early stage of training (the first 10 rounds), the accuracy of each model is low, the cross entropy loss is high, and the method is in a leading position; in the middle stage of training (10 to 35 rounds), the accuracy of each model gradually improves. The loss value is calculated using the cross entropy loss function and the parameters of the model are optimized, but the accuracy of the model of the present invention begins to approach stability; in the later stage of training (35 to 60 rounds), the accuracy of each model has stabilized. The curves of the traditional CNN network and LSTM network fluctuate greatly and only begin to converge in the 50th Epoch. The curve of the improved GCN model is relatively flat and fluctuates less. The BALG model proposed in the present invention not only converges faster, but also has a stable training process and high accuracy, and has always been in a leading position in the subsequent training process, further reflecting the advantages of the BALG model.

[0035] When using the causal relationship method based on BERT semantic enhancement to extract causal relationships from data sets in the financial field, the effect of pre-training with BERT in combination with the causal relationship candidate word library constructed in this paper is better. This shows that LeakGAN has great advantages over ordinary CCN algorithms in text in the financial field.

[0036] In addition, the above embodiments are only used to illustrate the specific embodiments of the present invention rather than to limit it. Those skilled in the art should understand that some of the technical features therein may be replaced by equivalents, and these modifications and replacements also fall within the protection scope of the present invention.

Claims

1. A causal relationship extraction method based on BERT semantic enhancement, including causal relationship candidate word library, BERT pre-training, and causal relationship extraction. It is characterized in that The proper nouns in various fields are learned through the causal relationship candidate word library, the features of the proper nouns are learned in the BERT pre-training, and the pre-trained word vectors are input into the Bi-LSTM network to extract text features. At the same time, multi-feature fusion is performed to learn more features; then, features are further extracted through the adversarial neural network; finally, the CRF is used for serialization output to realize the extraction of causal relationships; the adversarial neural network is composed of the BERT pre-trained word vector layer, the Bi-LSTM layer, the LeakGAN layer and the softmax output classification layer; The causal relationship extraction method specifically comprises the following steps: 1) Data preprocessing. The main tasks of data preprocessing include two aspects: one is to preliminarily screen the content of the text, delete the sentence components by default or unify the sentence format; the other is to annotate the screened sentences; 2) BERT pre-training, pre-training for both the causal enhancement model and the basic model; 3) Feature fusion based on causal relationships: In addition to basic position features, in order to select more accurately, the relevance of words is considered to measure the relevance between causal words and context words, the sentiment intensity of words, and the weight of the words themselves; 4) Feature learning for causal relationship extraction. Using BALG means learning the semantic features of complete sentences through the BiLSTM bidirectional recurrent network, adding a word-level attention layer to focus the feature vector on words that play a key role in causal relationship extraction, and LeakGAN implements text adversarial learning to learn high-discrimination features. Among them, LeakGAN generates a model of adversarial network. The Manager module in the generator receives the feature vector from the discriminator and generates the target embedding and Worker module. 5) Causal relationship serialization output, using CRF for serialization annotation, annotating each word in the sentence and extracting the cause phrases and results.

2. The method according to claim 1, It is characterized in that A method combining causal prior knowledge with semantic data was used to extract causal candidate words from review texts in various fields. During the BERT pre-training process, the three extracted candidate words with the highest matching degree were selected and passed into BERT to learn complex proper nouns in sentences.

3. The method according to claim 1, It is characterized in that During BERT pre-training, each entity is indicated by a special mark # before and after. 15% of the words in a sentence are randomly selected for prediction. In 80% of cases, a special symbol [MASK] is used to replace it. In 10% of cases, an arbitrary word is used to replace it. In the remaining 10%, the original word remains unchanged. When predicting a word, the model relies more on contextual information to predict the word and gives the model a certain error correction capability.

4. The method according to claim 1, It is characterized in that When it comes to sequence labeling, punctuation marks are also labeled like words, marked as "O", and causal connectives are not labeled. Causal relationship extraction is not limited to marked explicit causality.

Images