A Relationship Extraction Method Based on BERT Language Model and Deep Local Features
By combining the BERT language model with deep local features, and utilizing multi-head self-attention and multi-layer convolutional neural networks, the problems of large scale, long training time and dataset in relation extraction of existing pre-trained models are solved, and more efficient relation extraction results are achieved.
Patent Information
- Application Number
- CN202410639729.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-22
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-05-22
AI Technical Summary
Existing pre-trained language models suffer from problems such as large model size, long training time, high dataset preparation cost, labeling errors, and uneven data distribution in relation extraction tasks, resulting in low accuracy when extracting complex relations.
By combining the BERT language model with deep local features, a deep local feature extraction module is constructed through multi-head self-attention and multi-layer convolutional neural networks. A gating mechanism is used to fuse feature vectors to improve the classification effect of relation extraction.
It improves the accuracy and efficiency of relation extraction, enabling a better understanding of contextual information, acquisition of complex local features, and generation of accurate classification vectors.
Smart Images

Figure CN118585787B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to relation extraction in the field of natural language processing, specifically a relation extraction method based on the BERT language model and deep local features, which can be widely applied to relation extraction in various fields. Background Technology
[0002] With the widespread use of artificial neural network models, significant progress has been made in text relation extraction. However, this technological advancement has also brought new problems and challenges. First, the scale of models is increasing. Researchers tend to use large models for relation extraction research. Compared to methods like CNNs and RNNs, pre-trained models have extremely large parameters, ranging from hundreds of millions to billions, resulting in lengthy training times. This poses difficulties for practical deployment and application. Second, there is the issue of datasets. Creating a suitable dataset for a specific task is very costly, requiring substantial manpower and time. For relation extraction datasets, the number ranges from thousands to hundreds of thousands of data points, inevitably containing issues such as labeling errors and uneven data distribution. These problems undoubtedly affect model performance.
[0003] While the accuracy of relation extraction tasks has improved significantly, researchers tend to use external tools once the accuracy reaches a certain level. For example, those with the resources may retrain their models on more suitable large-scale corpora or use dependency parsers. Moreover, the model's accuracy remains low when the sample has a large number of words and complex relationships. For instance, the widely recognized Tacred dataset only scores around 70 points on the evaluation metric, and the DocRED dataset, a large document-level relation extraction dataset, only achieves a maximum score of around 60 points. When using pre-trained models, researchers typically use pre-trained models like BERT as encoders to encode text, requiring only fine-tuning in downstream tasks. However, this approach raises questions: will using corpora more suitable for downstream tasks yield better results? Will fine-tuning negatively impact the pre-trained model? These are all issues that need to be explored. To address these concerns, this invention proposes a relation extraction method based on the BERT language model and deep local features. By constructing a deep local feature extraction module composed of multi-head self-attention and multi-layer convolutional neural networks downstream of BERT, the classification performance of relation extraction is further improved. Summary of the Invention
[0004] This invention discloses a relation extraction method based on the BERT language model and deep local features, which can further improve the classification effect of relation extraction. The method is characterized by the following steps:
[0005] S1. The sentences from which the relations to be extracted are preprocessed using the R-BERT method, and the preprocessed sentences are fed into a BERT language model for encoding to obtain the BERT encoding of the sentences;
[0006] S2. Calculate the average value of the hidden states of the sub-word sequences of target entity 1 and target entity 2 in the BERT encoding of the sentence to obtain the hidden codes of target entity 1 and target entity 2. Then, process the hidden codes of target entity 1 and target entity 2, as well as the hidden states of the classifier in the BERT encoding of the sentence, with activation functions and linear layers to obtain the feature vectors of target entity 1, target entity 2 and classifier.
[0007] S3. The BERT encoding of the sentence is fed into a deep local feature extraction module consisting of multi-head self-attention and multi-layer convolutional neural networks to obtain the local feature vector of the sentence;
[0008] S4. The feature vectors of target entity 1, target entity 2 and classifier, as well as the local feature vector of the sentence, are fed into a feature vector fusion layer composed of a linear layer and a gating layer to obtain the classification vector of the sentence. Then the classification vector of the sentence is fed into the classification layer to obtain the category label of the relationship between target entity 1 and target entity 2 in the sentence.
[0009] The BERT neural network refers to the Bidirectional Encoder Representations from Transformers (BERT) language model proposed by Google AI Language in the paper "Devlin J, Chang MW, Lee K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding[J]. arXiv preprint arXiv:1810.04805,2018".
[0010] The R-BERT method refers to a relation extraction and processing method based on BERT proposed by Wu and He in the paper "Wu S, He Y. Enriching pre-trained language model with entity information for relation classification[C] / / Proceedings of the 28th ACM international conference on information and knowledge management.2019:2361-2364."
[0011] Furthermore, step S1 specifically includes:
[0012] S1.1 Process the sentence S containing two target entities e1 and e2 according to the R-BERT method. Insert a special marker "$" at the beginning and end of the first target entity e1, and insert a special marker "#" at the beginning and end of the second target entity e2, to obtain the preprocessed sentence S′.
[0013] S1.2 The preprocessed sentence S′ is fed into a BERT language model for encoding, resulting in the BERT encoding H of sentence S′ shown below. s :
[0014]
[0015] Where j1 and k1 are the start and end positions of target entity e1 in sentence S′, respectively, and j2 and k2 are the start and end positions of target entity e2 in sentence S′, respectively. These are the first and last words of the target entity e1 in the BERT encoding H. s Hidden states in These are the first and last subwords of the target entity e2 in the BERT encoding H. s The hidden state in the BERT language model is n, which is the number of sub-words segmented from sentence S′ by the BERT language model, and d is the dimension of the hidden state in the BERT language model.
[0016] Furthermore, step S2 specifically includes:
[0017] S2.1 The word sequences of target entity e1 and target entity e2 are processed in BERT encoding H respectively. s The hidden states are averaged and then fed into a hyperbolic tangent activation function and a linear layer liner1 for processing to obtain the feature vectors of e1 and e2. The calculation process is as follows:
[0018]
[0019]
[0020] Where tanh(·) denotes the hyperbolic tangent activation function, W1∈R d×d b1∈R d These represent the weights and bias parameters of the linear layer liner1, respectively. Encoding H for BERT s The t-th hidden state in the data;
[0021] S2.2 Hidden state of classifier [CLS] in sentence BERT encoding The vectors are fed into a hyperbolic tangent activation function and a linear layer liner2 for processing, resulting in the feature vector of the classifier [CLS]. The calculation process is as follows:
[0022]
[0023] Where W2∈R d×d b2∈R d These represent the weights and bias parameters of the linear layer liner2, respectively.
[0024] Furthermore, step S3 specifically includes:
[0025] S3.1 Encode the sentence using BERT H s Feed the input into a multi-head attention module to obtain the attention encoding H of the sentence. m The calculation process is as follows:
[0026] H m =MHSA(H s )∈R n×d (5)
[0027] Wherein, MHSA(·) represents a scaled dot product multi-head self-attention, which is proposed in the paper "Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[J].Advancesin neural information processing systems,2017,30.";
[0028] S3.2 Encode the sentence's attention H mThe input is fed into an L-layer convolutional neural network for deep local feature extraction, yielding the output of the L-th layer. The computation process of the L-th layer is shown below:
[0029]
[0030] Where δ(·) represents the ReLU activation function, Let be the parameters of the convolutional kernel in the l-th layer of the convolutional neural network, where d1 = d is the number of convolutional kernels, d2 = d is the number of input channels, and e represents the window width of the convolutional kernel. Let be the bias parameters in the l-th layer of the convolutional neural network. Let l be the input to the l-th layer of the convolutional neural network, and when l = 1, we have This is the output of the l-th layer of the convolutional neural network, and when l = L This represents the output of the Lth layer of the convolutional neural network;
[0031] S3.3 outputs the Lth layer of the convolutional neural network Perform max pooling to obtain pooling vector The BERT encoding of sentence S′ is H s Perform average pooling to obtain H s The pooling vector v s Then With v s By performing a point-by-point addition residual connection, the local feature vector of the sentence is obtained. The calculation process is as follows:
[0032]
[0033] v s =AvePooling(H s )∈R d (8)
[0034]
[0035] In this context, MaxPooling(·) represents the average pooling operation, AvePooling(·) represents the average pooling operation, and "+" represents point-by-point addition;
[0036] Furthermore, step S4 specifically includes:
[0037] S4.1 The feature vectors of the two target entities e1 and e2 and Concatenate them separately into the local feature vectors and the feature vector of the classifier [CLS] Following this, a concatenated feature vector is formed. and The calculation process is as follows:
[0038]
[0039]
[0040] Where [:] denotes the vector concatenation operation;
[0041] S4.2 will concatenate the feature vectors. and The vectors are fed into linear layers liner3 and liner4 respectively for linear transformation to obtain the global feature vector of the sentence. and And and Concatenate into a combined feature vector The calculation process is as follows:
[0042]
[0043]
[0044]
[0045] Where W3∈R d×3d b3∈R d These are the weights and bias parameters of the linear layer liner3, where W4∈R. d×3d b4∈R d These are the weights and bias parameters of the linear layer liner4, respectively;
[0046] S4.3 uses combined feature vectors respectively Initialize and update the gate state g of gate1 and reset the gate state g of gate2 u and g r The calculation process is as follows:
[0047]
[0048]
[0049] Among them, W u ∈R d×2d b u ∈R d These are the weights and biases of gate1, respectively, W. r ∈R d×2d b r ∈R dThese represent the weights and biases of gate2, respectively, and sigmoid(·) represents the sigmoid activation function;
[0050] S4.4 Based on the gating state g u and g r Calculate the candidate vector H of the feature vector fusion layer respectively. can ∈R d And the final classification vector Z∈R d The calculation process is as follows:
[0051]
[0052]
[0053] Where tanh(·) represents the hyperbolic tangent activation function, ⊙ represents pointwise multiplication, and W h ∈R d b h ∈R d These represent the weights and biases of a linear layer;
[0054] S4.5 feeds the classification vector Z into the classification layer to obtain the category labels of the relationship between target entities e1 and e2 in sentence S. The calculation process is as follows:
[0055] o = W o Z+b o (19)
[0056] p = Softmax(o) (20)
[0057]
[0058] Where Ω represents the label set of relation categories, o∈R|Ω| is the confidence vector of the classification vector Z on the label set Ω, |Ω| is the number of labels in the label set Ω, and W o ∈R |Ω|×d b o ∈R |Ω| These represent the weights and biases in the classification layer, p∈R |Ω| Let represent the probability distribution vector of the relation extracted from sentence S on the label set Ω, and let Softmax(·) denote the normalization exponential function in the neural network. Let p(y) represent the label y in Ω that maximizes p(y), where p(y) represents the probability of label y in p, and y is any label in Ω.
[0059] The present invention has the following advantages:
[0060] (1) Use the BERT language model as an encoder so that the same word can represent different semantics in different sentence contexts, so that the downstream network can extract text features more accurately.
[0061] (2) A deep local feature extraction module consisting of multi-head self-attention and multi-layer convolutional neural networks is proposed. The multi-head self-attention mechanism allows the model to better understand the context information, and the multi-layer convolutional neural network can acquire local features from simple to complex. The combination of the two can obtain deep semantic understanding and fine-grained features, and construct a richer and more hierarchical representation of sentence local features.
[0062] (3) In order to integrate different feature vectors, a gating mechanism-based processing method was designed. By designing update gates and reset gates, the contribution of different feature vectors is measured to generate accurate classification vectors. Attached Figure Description
[0063] Figure 1 This is the relationship extraction model between the BERT language model and deep local features in the embodiment;
[0064] Figure 2 This refers to the deep local feature extraction module in the embodiment.
[0065] Figure 3 This is the feature vector fusion layer in the embodiment. Detailed Implementation
[0066] The present invention will be further described below with reference to specific embodiments, but the scope of protection of the present invention is not limited to the following embodiments.
[0067] For any sentence whose relation to be extracted, according to Figure 1 The model of the present invention, as shown, classifies the relationship between target entity 1 and target entity 2 in a sentence through the following steps:
[0068] S1. The sentences from which the relations to be extracted are preprocessed using the R-BERT method, and the preprocessed sentences are fed into a BERT language model for encoding to obtain the BERT encoding of the sentences;
[0069] S2. Calculate the average value of the hidden states of the sub-word sequences of target entity 1 and target entity 2 in the BERT encoding of the sentence to obtain the hidden codes of target entity 1 and target entity 2. Then, process the hidden codes of target entity 1 and target entity 2, as well as the hidden states of the classifier in the BERT encoding of the sentence, with activation functions and linear layers to obtain the feature vectors of target entity 1, target entity 2 and classifier.
[0070] S3. The BERT encoding of the sentence is fed into a deep local feature extraction module consisting of multi-head self-attention and multi-layer convolutional neural networks to obtain the local feature vector of the sentence;
[0071] S4. The feature vectors of target entity 1, target entity 2 and classifier, as well as the local feature vector of the sentence, are fed into a feature vector fusion layer composed of a linear layer and a gating layer to obtain the classification vector of the sentence. Then the classification vector of the sentence is fed into the classification layer to obtain the category label of the relationship between target entity 1 and target entity 2 in the sentence.
[0072] The BERT neural network refers to the Bidirectional Encoder Representations from Transformers (BERT) language model proposed by Google AI Language in the paper "Devlin J, Chang MW, Lee K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding[J]. arXiv preprint arXiv:1810.04805,2018".
[0073] The R-BERT method refers to a relation extraction and processing method based on BERT proposed by Wu and He in the paper "Wu S, He Y. Enriching pre-trained language model with entity information for relation classification[C] / / Proceedings of the 28th ACM international conference on information and knowledge management.2019:2361-2364."
[0074] Furthermore, step S1 specifically includes:
[0075] S1.1 Process the sentence S containing two target entities e1 and e2 according to the R-BERT method. Insert a special marker "$" at the beginning and end of the first target entity e1, and insert a special marker "#" at the beginning and end of the second target entity e2, to obtain the preprocessed sentence S′.
[0076] S1.2 The preprocessed sentence S′ is fed into a BERT language model for encoding, resulting in the BERT encoding H of sentence S′ shown below. s :
[0077]
[0078] Where j1 and k1 are the start and end positions of target entity e1 in sentence S′, respectively, and j2 and k2 are the start and end positions of target entity e2 in sentence S′, respectively. These are the first and last words of the target entity e1 in the BERT encoding H. s Hidden states in These are the first and last subwords of the target entity e2 in the BERT encoding H. s The hidden state in the BERT language model is n, which is the number of sub-words segmented from sentence S′ by the BERT language model, and d is the dimension of the hidden state in the BERT language model.
[0079] Furthermore, step S2 specifically includes:
[0080] S2.1 The word sequences of target entity e1 and target entity e2 are processed in BERT encoding H respectively. s The hidden states are averaged and then fed into a hyperbolic tangent activation function and a linear layer liner1 for processing to obtain the feature vectors of e1 and e2. The calculation process is as follows:
[0081]
[0082]
[0083] Where tanh(·) denotes the hyperbolic tangent activation function, W1∈R d×d b1∈R d These represent the weights and bias parameters of the linear layer liner1, respectively. Encoding H for BERT s The t-th hidden state in the data;
[0084] S2.2 Hidden state of classifier [CLS] in sentence BERT encoding The vectors are fed into a hyperbolic tangent activation function and a linear layer liner2 for processing, resulting in the feature vector of the classifier [CLS]. The calculation process is as follows:
[0085]
[0086] Where W2∈R d×d b2∈Rd These represent the weights and bias parameters of the linear layer liner2, respectively.
[0087] Furthermore, step S3 specifically includes:
[0088] S3.1 Encode the sentence using BERT H s Feed the input into a multi-head attention module to obtain the attention encoding H of the sentence. m The calculation process is as follows:
[0089] H m =MHSA(H s )∈R n×d (5)
[0090] Wherein, MHSA(·) represents a scaled dot product multi-head self-attention, which is proposed in the paper "Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[J].Advancesin neural information processing systems,2017,30.";
[0091] S3.2 Encode the sentence's attention H m The input is fed into an L-layer convolutional neural network for deep local feature extraction, yielding the output of the L-th layer. The computation process of the L-th layer is shown below:
[0092]
[0093] Where δ(·) represents the ReLU activation function, Let be the parameters of the convolutional kernel in the l-th layer of the convolutional neural network, where d1 = d is the number of convolutional kernels, d2 = d is the number of input channels, and e represents the window width of the convolutional kernel. These are the bias parameters in the l-th layer of the convolutional neural network. Let be the input of the l-th layer of the convolutional neural network, and when l = 1, we have This is the output of the l-th layer of the convolutional neural network, and when l = L This represents the output of the Lth layer of the convolutional neural network;
[0094] S3.3 outputs the Lth layer of the convolutional neural network Perform max pooling to obtain pooling vector The BERT encoding of sentence S′ is H s Perform average pooling to obtain H sThe pooling vector v s Then With v s By performing a point-by-point addition residual connection, the local feature vector of the sentence is obtained. The calculation process is as follows:
[0095]
[0096] v s =AvePooling(H s )∈R d (8)
[0097]
[0098] In this context, MaxPooling(·) represents the average pooling operation, AvePooling(·) represents the average pooling operation, and "+" represents point-by-point addition;
[0099] Furthermore, step S4 specifically includes:
[0100] S4.1 The feature vectors of the two target entities e1 and e2 and Concatenate them separately into the local feature vectors and the feature vector of the classifier [CLS] Following this, a concatenated feature vector is formed. and The calculation process is as follows:
[0101]
[0102]
[0103] Where [:] denotes the vector concatenation operation;
[0104] S4.2 will concatenate the feature vectors. and The vectors are fed into linear layers liner3 and liner4 respectively for linear transformation to obtain the global feature vector of the sentence. and And and Concatenate into a combined feature vector The calculation process is as follows:
[0105]
[0106]
[0107]
[0108] Where W3∈R d×3d b3∈R d These are the weights and bias parameters of the linear layer liner3, where W4∈R. d×3d b4∈R d These are the weights and bias parameters of the linear layer liner4, respectively;
[0109] S4.3 uses combined feature vectors respectively Initialize and update the gate state g of gate1 and reset the gate state g of gate2 u and g r The calculation process is as follows:
[0110]
[0111]
[0112] Among them, W u ∈R d×2d b u ∈R d These are the weights and biases of gate1, respectively, W. r ∈R d×2d b r ∈R d These represent the weights and biases of gate2, respectively, and sigmoid(·) represents the sigmoid activation function;
[0113] S4.4 Based on the gating state g u and g r Calculate the candidate vector H of the feature vector fusion layer respectively. can ∈R d And the final classification vector Z∈R d The calculation process is as follows:
[0114]
[0115]
[0116] Where tanh(·) represents the hyperbolic tangent activation function, ⊙ represents pointwise multiplication, and W h ∈R d b h ∈R d These represent the weights and biases of a linear layer;
[0117] S4.5 feeds the classification vector Z into the classification layer to obtain the category labels of the relationship between target entities e1 and e2 in sentence S. The calculation process is as follows:
[0118] o = Wo Z+b o (19)
[0119] p = Softmax(o) (20)
[0120]
[0121] Where Ω represents the label set of relation categories, o∈R|Ω| is the confidence vector of the classification vector Z on the label set Ω, |Ω| is the number of labels in the label set Ω, and W o ∈R |Ω|×d b o ∈R |Ω| These represent the weights and biases in the classification layer, p∈R |Ω| Let represent the probability distribution vector of the relation extracted from sentence S on the label set Ω, and let Softmax(·) denote the normalization exponential function in the neural network. Let p(y) represent the label y in Ω that maximizes p(y), where p(y) represents the probability of label y in p, and y is any label in Ω.
[0122] Furthermore, Figure 1 The model of this invention shown below is trained using the cross-entropy loss function as follows:
[0123]
[0124] Among them, y i It is the i-th element in the actual tag.
[0125] Application Examples
[0126] 1. Instance Environment
[0127] The hyperparameter settings for this example model are shown in Table 1.
[0128] Table 1: Hyperparameters of the instance
[0129]
[0130]
[0131] 2. Dataset
[0132] The dataset used in this experiment is the Stanford MOOC Post dataset proposed by Agrawal et al., which can be downloaded from http: / / datastage.stanford.edu / StanfordMoocPosts. This dataset contains 29,604 forum posts, collected from 11 public online courses at Stanford University, divided into three different fields: humanities, medicine, and education. All posts were categorized by Agrawal et al. into six dimensions: opinion, question, answer, sentiment, confusion, and urgency, and were also rated. Opinions, questions, and answers were rated 0 and 1, respectively, while sentiment, confusion, and urgency were rated from 1 to 7, as shown in Table 2.
[0133] Table 2. Sample distribution of the SemEval-2010 Task 8 dataset.
[0134]
[0135]
[0136] This experiment uses the publicly available dataset SemEval-2010 Task8 proposed in the paper "HENDRICKX I,KIM SN,KOZAREVAZ,et al.Semeval-2010task8:Multi-way classification of semantic relations between pairs of nominals[J].arXiv preprint arXiv:191110422,2019.", as shown in Table 2.
[0137] 3. Case Comparison Results
[0138] This example is based on R-BERT experiments, and the results are shown in Table 3. (B) represents the base version of BERT, containing 12 Transformer encoder layers with a total of 110M parameters, an input layer embedding vector dimension of 768, and a hidden layer dimension of 768. (L) represents the enhanced version of BERT, containing 24 Transformer encoder layers with a total of 340M parameters, and both the input layer embedding vector dimension and the hidden layer dimension are 1024. R-BERT+MSF+GF represents the method used in this example, while R-BERT represents the method used in the paper "Wu S, He Y. Enriching pre-trained language model with entity information for relation classification[C] / / Proceedings of the 28th ACM international conference on information and knowledge management.2019:2361-2364."
[0139] Table 3 Experimental Results
[0140]
[0141] Note: "*" indicates that the experimental results are from the original paper, and "+" indicates that the experimental results are from the experiment in this example.
[0142] The results in Table 3 show that the model proposed in this invention, implemented in this example, significantly outperforms the R-BERT original model in both the base version of BERT and the Macro-F1 of the base enhanced version of BERT. This fully demonstrates that the relationship extraction method based on the BERT language model and deep local features proposed in this invention is feasible and excellent.
Claims
1. A method for extracting the relationship between BERT language model and deep local features, characterized in that... Includes the following steps: S1. The sentences from which the relations to be extracted are preprocessed using the R-BERT method, and the preprocessed sentences are fed into a BERT language model for encoding to obtain the BERT encoding of the sentences; S2. Calculate the average value of the hidden states of the sub-word sequences of target entity 1 and target entity 2 in the BERT encoding of the sentence to obtain the hidden codes of target entity 1 and target entity 2. Then, process the hidden codes of target entity 1 and target entity 2, as well as the hidden states of the classifier in the BERT encoding of the sentence, with activation functions and linear layers to obtain the feature vectors of target entity 1, target entity 2 and classifier. S3. The BERT encoding of the sentence is fed into a deep local feature extraction module consisting of multi-head self-attention and multi-layer convolutional neural networks to obtain the local feature vector of the sentence; S4. The feature vectors of target entity 1, target entity 2 and classifier, as well as the local feature vector of the sentence, are fed into a feature vector fusion layer composed of a linear layer and a gating layer to obtain the classification vector of the sentence. Then the classification vector of the sentence is fed into the classification layer to obtain the category label of the relationship between target entity 1 and target entity 2 in the sentence. Step S1 specifically includes: S1.1 processes the sentence S containing two target entities e1 and e2 according to the R-BERT method, inserting a special marker "$" at the beginning and end of the first target entity e1, and a special marker "#" at the beginning and end of the second target entity e2, to obtain the preprocessed sentence. ; S1.2 will preprocess the sentence The data is fed into a BERT language model for encoding, resulting in the sentence shown below. BERT encoding : (1) Where j1 and k1 are the target entity e1 in the sentence The start and end positions in the sentence, j2 and k2 are the target entity e2 in the sentence. The start and end positions in the text , These are the first and last words of the target entity e1 encoded in BERT. Hidden states in , These are the first and last subwords of the target entity e2 encoded in BERT. Hidden states in For sentences The number of subwords segmented by the BERT language model This represents the dimension of the hidden states in the BERT language model. Step S2 specifically includes: S2.1 Encode the word sequences of target entity e1 and target entity e2 in BERT respectively. The hidden states are averaged and then fed into a hyperbolic tangent activation function and a linear layer (linear1) to obtain the feature vectors e1 and e2. , The calculation process is as follows: in, This represents the hyperbolic tangent activation function. , These represent the weights and bias parameters of the linear layer (linear1), respectively. Encoding for BERT The first in One hidden state; S2.2 Hidden state of classifier [CLS] in sentence BERT encoding The vectors are fed into a hyperbolic tangent activation function and a linear layer (linear2) for processing to obtain the feature vector of the classifier [CLS]. The calculation process is as follows: (4) in, , These represent the weights and bias parameters of the linear layer 2, respectively. Step S3 specifically includes: S3.1 BERT encoding of the sentence Feed the input into a multi-head attention module to obtain the attention encoding of the sentence. The calculation process is as follows: (5) in, This indicates scaling the dot product with multi-head self-attention. S3.2 Encoding sentence attention The input is fed into an L-layer convolutional neural network for deep local feature extraction, yielding the output of the L-th layer convolutional neural network, where the L-th layer is... The computation process of a layered convolutional neural network is shown below: (6) in, Represents the ReLU activation function. For the first The parameters of the convolution kernel in a multilayer convolutional neural network, and have The number of convolution kernels, Input the number of channels. This represents the window width of the convolution kernel. For the first Bias parameters in a multilayer convolutional neural network For the first The input of a layer convolutional neural network, and when From time to time , For the first The output of a layer convolutional neural network, and when hour This represents the output of the Lth layer of the convolutional neural network; S3.3 outputs the Lth layer of the convolutional neural network Perform max pooling to obtain pooling vector , the sentence BERT encoding Perform average pooling to obtain pooling vector Then and By performing a point-by-point addition residual connection, the local feature vector of the sentence is obtained. The calculation process is as follows: (7) (8) (9) in, This indicates a max pooling operation. This indicates the average pooling operation. " indicates addition point by point; Step S4 specifically includes: S4.1 will combine the two target entities and eigenvectors and Concatenate them separately into the local feature vectors and the feature vector of the classifier [CLS] Following this, a concatenated feature vector is formed. and The calculation process is as follows: (10) (11) in, This indicates the operation of concatenating vectors. S4.2 will concatenate the feature vectors. and The vectors are fed into linear layers linear3 and linear4 respectively for linear transformation to obtain the global feature vector of the sentence. and and will and Concatenate into a combined feature vector The calculation process is as follows: (12) (13) (14) in, , These are the weights and bias parameters of the linear layer (linear3), respectively. , These are the weights and bias parameters of the linear layer (linear4), respectively. S4.3 Using combined feature vectors respectively Initialize and update the gate state of gate1 and reset the gate state of gate2. and The calculation process is as follows: (15) (16) in, , These update the weights and biases of gate1, respectively. , These are the weights and biases for resetting gate2, respectively. This represents the sigmoid activation function; S4.4 Based on the gating state and Calculate the candidate vectors of the feature vector fusion layer respectively. and the final classification vector The calculation process is as follows: (17) (18) in, This represents the activation function of the hyperbolic tangent function. This indicates point-by-point multiplication. , These are the weights and biases of a linear layer, respectively. S4.5 will classify the vectors The data is fed into a classification layer to obtain the category labels representing the relationship between target entities e1 and e2 in sentence S. The calculation process is as follows: (19) (20) in, A set of labels representing relation categories For classification vectors In the tag set The confidence vector on, For a set of tags The number of tags in , These represent the weights and biases in the classification layer, respectively. Relation extraction for sentence S from the label set The probability distribution vector on, This represents the normalized exponential function in a neural network. Expressing the requirement to make The maximum value Chinese label , express Chinese label The probability, for Any tag in the text.
