A Denoising Method and Model for Remotely Supervised Relation Extraction Based on BERT Entity Awareness
By employing BERT entity-aware sentence encoding and generative adversarial learning, the noise problem in remote supervised relation extraction is solved, improving the accuracy of noise filtering and classification, simplifying the model structure, and enhancing processing efficiency.
Patent Information
- Application Number
- CN202310385105.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-11
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-04-11
AI Technical Summary
Existing relation extraction methods based on remote supervision contain noisy data, which leads to a decline in model performance, especially the noise problems of false positives and false negatives. The existing technology uses PCNN encoding, which has low efficiency and a large number of parameters, resulting in poor classification results.
We employ a BERT entity-aware sentence encoding method to encode sentences. We use the pre-trained language model BERT to obtain feature information, and perform linear transformation on the hidden state vectors of sentence header tags, header target entities, and tail target entities. Combined with selective attention mechanism and generative adversarial learning, we filter and classify noisy data.
It improves the accuracy of noise filtering and classification, simplifies the model structure, increases processing efficiency, and reduces the impact of noisy data on model performance.
Smart Images

Figure CN116796753B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of natural language processing, and in particular to a noise reduction method and model for remote supervised relation extraction based on BERT entity perception. Background Technology
[0002] Natural Language Processing (NLP) is an important field in computer science and artificial intelligence. It primarily studies the theories and methods for effective communication and interaction between humans and computers using natural language. Information extraction is a crucial task in NLP, involving the extraction of specific event or factual information from natural language text to automatically classify, extract, and reconstruct text data. Information extraction has three sub-tasks: relation extraction, event extraction, and named entity recognition. Relation extraction plays a vital role in information extraction, referring to inferring the relationships between entity pairs from given plain text based on sentence semantic information. For example, in the sentence "Chengdu is the capital city of Sichuan," relation extraction extracts the relationship between "Chengdu" and "Sichuan" and constructs a triple (Chengdu-Capital City-Sichuan), which is then used for downstream tasks such as knowledge graphs, question answering systems, text summarization, and machine translation.
[0003] Existing relation extraction tasks are categorized into supervised learning-based methods and remote supervision-based methods based on label type. Supervised learning-based methods can achieve higher accuracy and more reliable model results, but their application is limited due to the scarcity and high cost of manually labeled data. Therefore, remote supervision-based methods have been proposed to automate relation extraction.
[0004] Relation extraction methods based on remote supervision automatically label unlabeled text data by aligning relation pairs in a knowledge graph, thereby generating a relation extraction dataset and alleviating the pressure of manual data labeling. For example, assuming the entities "Xiaoming" and "enterprise" have the relation "employment", and given three sentences with unlabeled relations, all containing "Xiaoming" and "enterprise" as entities, then based on the remote supervision assumption, they would be labeled as representing the relation "employment".
[0005] However, due to its strong constraint assumption (i.e., all sentences containing the same entity pairs are labeled as the same relation in the knowledge graph), mislabeling problems arise during dataset construction. For example, three unlabeled sentences containing "Xiaoming" and "enterprise" with the relation "employ" are all assigned the relation "employ" because the entity pairs correspond. However, since there can be multiple relations between two entities in different contexts, only one sentence actually has the correct relation, while the other sentences' entity pairs do not have the relation "employ." Therefore, relation extraction methods based on remote supervision are prone to such mislabeled sentences, known as false positive (FP) noise. Furthermore, according to the remote supervision assumption, when a sentence's entity pair does not match existing entity pairs, it is labeled "NA," i.e., unknown. However, the entity pairs in this sentence may actually have a relation. Therefore, relation extraction methods based on remote supervision are prone to labeling sentences with a relation as "NA," i.e., unknown, simply because of the incompleteness of existing data (also known as false negative (FN) noise). Therefore, the relation extraction method based on remote supervision has mislabeled datasets, and the resulting noise will reduce the performance of the model.
[0006] Currently, there are methods for processing noisy data obtained after relation extraction based on remote supervision. For example, in the scheme disclosed in "Kailong Hao, Botao Yu, Wei Hu. Knowing False Negatives-An Adversarial Training Method for Distantly Supervised Relation Extraction.[C], Empirical Methods in Natural Language Processing, 2021, 2021.emnlp-main:9661-9672.", the sentence is encoded using PCNN to obtain sentence features, and then the dataset is filtered through the memory mechanism of deep neural networks (specifically, using deep neural networks to first learn clean samples without noisy data, and then learn noisy samples) to obtain FN noisy sentences. Finally, a classifier is trained by training generative adversarial methods and contrastive learning methods to reassign a correct label to the FN noisy sentences. However, in the aforementioned existing technologies, firstly, the memory mechanism of its deep neural network has not been proven to be effective. Therefore, the effectiveness of this memory mechanism in mining FN noise is questionable. Secondly, it uses PCNN for sentence encoding, which is also a commonly used sentence encoding method in existing technologies. PCNN captures sentence features through convolution, which is inefficient. Furthermore, the data obtained after segmentation and concatenation after convolution has high dimensionality. Adding fully connected layers will result in a large number of parameters, complex model structure, low efficiency, and long processing time for the entire model. Moreover, its pooling layers will lose a lot of valuable information, resulting in poor label classification performance for FN noisy sentences. Summary of the Invention
[0007] Based on this, the purpose of this invention is to provide a noise reduction method for remote supervised relation extraction based on BERT entity perception, which can extract sentence information quickly and accurately, and improve the subsequent noise filtering and classification effects.
[0008] A noise reduction method for remote supervised relation extraction based on BERT entity perception includes the following steps:
[0009] S11. The sentence is encoded using a BERT entity-aware sentence encoding method to obtain a sentence encoding vector containing entity pair information.
[0010] The BERT entity-aware sentence encoding method includes: using a pre-trained language model BERT to embed words into the sentence, and obtaining the output vector representations of the head target entity, the tail target entity, and the sentence head marker based on the hidden state vectors of the head target entity, the tail target entity, and the sentence head marker obtained by BERT; and then obtaining sentence encoding vector one based on the output vector representations of the head target entity, the tail target entity, and the sentence head marker.
[0011] The sentence is noise data labeled NA obtained after extracting the remote supervision relation;
[0012] S12. Based on the sentence encoding vector one, obtain the sentence packet representation vector one of the sentence packets composed of sentences containing the same entity pair;
[0013] S13. Obtain the relation representation value of the sentence packet representation vector one according to the sentence packet representation vector one;
[0014] S15. Based on the sentence relation representation of the sentence package obtained from the relation representation value, obtain the FN noise dataset formed by sentences with potential semantic value;
[0015] S2. Classify the sentence relationships in the FN noise dataset.
[0016] In this scheme, for the noisy data labeled NA obtained after extracting remote supervision relations, the sentence encoding method abandons the existing PCNN approach and instead adopts a BERT entity-aware sentence encoding method. Specifically, using the pre-trained BERT language model as a word embedding yields good feature information, improving the accuracy of sentence information representation and thus enhancing the sentence encoding performance, which in turn improves subsequent noise filtering and classification. Furthermore, the hidden state vectors of the sentence head label (CLS), head target entity, and tail target entity obtained after BERT processing are processed to obtain the sentence vector representation. This processing method is simple and efficient. Moreover, the introduction of head and tail target entities as supplements to the CLS label in the BERT language model fully leverages the advantages of BERT.
[0017] Further, in step S11, the sentence encoding vector is represented as follows:
[0018] v=W3[concat(H0′,1′,2′)]+3
[0019] Where v represents the sentence encoding vector, concat represents the concatenation of vectors, H0′ represents the output vector representation of the sentence header marker, H1′ represents the output vector representation of the header target entity, H2′ represents the output vector representation of the tail target entity, and W3 and b3 are both trainable parameters.
[0020] Furthermore, the output vector representations of the head target entity and the tail target entity are obtained using the following formula:
[0021]
[0022]
[0023] Where W1, W2, b1, and b2 are all trainable parameters, and the vector H... i To H j H represents the hidden state vector obtained by BERT for the head target entity. k To H m denoted as the hidden state vector of the tail target entity obtained through BERT, and tanh is the hyperbolic tangent activation function.
[0024] Furthermore, the output vector representation of the sentence header tag is obtained through the following formula:
[0025] H′0=W0[tanh(H0)]+b0
[0026] Here, H0 represents the hidden state vector of the sentence header obtained by BERT, and W0 and b0 are trainable parameters.
[0027] Further, in step S12, the sentence packet representation vector is represented as follows:
[0028]
[0029] Where g represents the sentence package, v represents vector one, and v represents vector one. i Let a represent the sentence encoding vector of the i-th sentence in the sentence packet. i This represents the weight of the i-th sentence obtained through the selective attention mechanism.
[0030] Furthermore, the formula for calculating the weight is as follows:
[0031]
[0032] Where N represents the total number of sentences in the sentence package, r j The sentence relation is represented by a vector, j∈(0,1).
[0033] Furthermore, in step S13, the formula for calculating the relation representation value is as follows:
[0034] P(r j |g i = softmax(W) c g i +b c )
[0035] Wherein, P(r) j |g i ) represents a sentence package representing a vector g i expression r j The value represented by the entity in the relation, W. c b c All of these are trainable parameters.
[0036] Further, step S2 includes:
[0037] S21. The sentences in the FN noise dataset are encoded using a BERT entity-aware sentence encoding method to obtain the sentence encoding vector 2 of the sentences in the FN noise dataset.
[0038] S22. Based on the second sentence encoding vector, obtain the second sentence packet representation vector of the sentence packet composed of sentences containing the same entity pair in the FN noise dataset;
[0039] S23. Obtain the relation representation value of the second sentence packet representation vector based on the second sentence packet representation vector;
[0040] S24. Based on the relation representation value of the sentence package representation vector two, obtain the entity pair relation of the sentence and label the entity pair relation of the sentence.
[0041] Furthermore, this invention also relates to a denoising model for remote supervised relation extraction based on BERT entity perception. This model includes a sub-model one and a sub-model two. Sub-model one filters the noise data labeled NA after remote supervised relation extraction to obtain an FN noise dataset. Sub-model two classifies the sentence relations in the FN noise dataset. Sub-model one includes a sentence encoding vector encoding module one, a sentence packet representation vector encoding module one, a relation representation value calculation module one, and an FN noise dataset acquisition module. The sentence encoding vector encoding module one encodes sentences using a BERT entity perception-based sentence encoding method to obtain a sentence encoding vector one containing entity pair information. The sentence packet representation vector encoding module one obtains a sentence packet representation vector one for sentences containing the same entity pair based on the sentence encoding vector one. The relation representation value calculation module one obtains the relation representation value of the sentence packet representation vector one based on the sentence packet representation vector one. The FN noise dataset acquisition module obtains an FN noise dataset composed of sentences with potential semantic value based on the sentence relation representation of the sentence packets obtained from the relation representation value.
[0042] Furthermore, the second sub-model includes a sentence encoding vector encoding module, a sentence packet representation vector encoding module, a relation representation value calculation module, and a sentence relation acquisition and annotation module. Specifically, the sentence encoding vector encoding module uses a BERT entity-aware sentence encoding method to encode sentences and obtain sentence encoding vectors for sentences in the FN noise dataset. The sentence packet representation vector encoding module is used to obtain sentence packet representation vectors for sentences containing the same entity pair in the FN noise dataset based on the sentence encoding vectors. The relation representation value calculation module is used to obtain the relation representation value of the sentence packet representation vectors based on the sentence packet representation vectors. The sentence relation acquisition and annotation module is used to obtain the entity pair relations expressed by the sentence based on the relation representation value of the sentence packet representation vectors and to annotate the entity pair relations of the sentence.
[0043] To better understand and implement this invention, the following detailed description is provided in conjunction with the accompanying drawings. Attached Figure Description
[0044] Figure 1 This is a schematic diagram of the module of the remote supervision relation extraction and noise reduction model based on BERT entity perception of the present invention.
[0045] Figure 2 This is a flowchart of the steps of the remote supervision relation extraction and noise reduction method based on BERT entity perception of the present invention.
[0046] Figure 3 This is a schematic diagram of the sentence encoding method based on BERT entity awareness according to the present invention. Detailed Implementation
[0047] In existing technologies, the processing of noisy data obtained after remote supervision relation extraction involves PCNN for sentence encoding. PCNN captures sentence features through convolution, which is inefficient. Furthermore, the high dimensionality of the data obtained after segmentation and concatenation following convolution, coupled with fully connected layers, leads to a large number of parameters, complex model structure, low efficiency, and long processing time. Additionally, its pooling layers lose a significant amount of valuable information, resulting in poor label classification of FN noisy sentences. Therefore, to improve the filtering and classification of noisy data obtained after remote supervision relation extraction, and to save processing time, this invention utilizes BERT entity-aware sentence encoding to quickly and accurately extract sentence information from noisy data labeled NA after remote supervision relation extraction, thereby improving subsequent noise filtering and classification performance.
[0048] Please see Figure 1-2 In this invention, a remote supervised relation extraction and denoising model based on BERT entity perception is provided. The model includes sub-model 1 and sub-model 2. Sub-model 1 is used to perform step S1, which is to filter the noise data obtained after remote supervised relation extraction and labeled as NA to obtain FN noise dataset with potential semantic value. Sub-model 2 is used to perform step S2, which is to classify the sentence relations of sentences in the FN noise dataset.
[0049] The sub-model 1 includes a sentence encoding vector encoding module 11, a sentence packet representation vector encoding module 12, a relation representation value calculation module 13, a sentence relation acquisition module 14, and an FN noise dataset acquisition module 15.
[0050] The sentence encoding vector encoding module 11 is used to execute step S11 to perform sentence encoding on the noise data obtained after the remote supervision relation extraction, thereby obtaining sentence encoding vector 1. The noise data is the data formed by the sentences labeled NA obtained from the remote supervision relation extraction.
[0051] Specifically, step S11 involves encoding the sentence to obtain a sentence encoding vector containing entity pair information.
[0052] This solution employs a BERT entity-aware sentence encoding method for sentence encoding. Please refer to [link / reference needed]. Figure 3 Specifically:
[0053] For a sentence with an entity pair, a start tag is first inserted on one side and an end tag on the other side of each entity in the pair. This allows the pre-trained language model BERT to capture the positional information of the two entities in the pair at the beginning and end of the entity pair. Additionally, tags are added to the beginning of the sentence. Then, word embeddings are performed using the pre-trained language model BERT.
[0054] In a specific implementation, for the target entity h with a head... i and tail target entity i The sentence s, in the head target entity h i Insert markers [E1] and [E1 / ] on both sides respectively, and place the target entity at the tail. i Insert [E2] and [E2 / ] on either side of the sentence, and add a sentence header marker [CLS] at the beginning of the sentence. For example, for a sentence with the target entities "kitchen" and "house", after inserting the markers, the sentence will become: "[CLS]The[E1]kitchen[E1 / ]is the last renovated part of the[E2]house[E2 / ]".
[0055] In this embodiment, for a target entity h containing a head i and tail target entity i Sentence s, assuming vector H i To H j It is the head target entity h i The final hidden state vector obtained through BERT, H k To H m The tail target entity i The final hidden state vector is obtained through BERT. In this embodiment, the head target entity h is obtained through the following linear transformation. i and tail target entity i The output vector representation is as follows:
[0056]
[0057]
[0058] Where H1′ represents the head target entity h i The output vector representation, H2′ represents the tail target entity. i The output vector representation, where W1, W2, b1, and b2 are all trainable parameters, W1 and W2 are matrices, and b1 and b2 are vectors, with W1 and W2 having the same matrix dimension and b1 and b2 having the same vector dimension, where W1 ∈ R. m×m ,2∈Rm×m R m×m In this context, m represents the size of the hidden state in BERT, and tanh represents the hyperbolic tangent activation function.
[0059] Furthermore, the output vector representation of the sentence header marker (i.e., [CLS]) is obtained through the following linear transformation:
[0060] H′0=W0[tanh(H0)]+b0 (3)
[0061] Where H′0 represents the output vector representation of the sentence header tags, H0 represents the final hidden state vector of the sentence header tags obtained by BERT, and W0 and b0 are trainable parameters, where W0 is a matrix and b0 is a vector. W0 has the same dimension as W1 and W1, i.e., W0∈R. m×m .
[0062] Furthermore, the sentence encoding vector one is obtained in the following way:
[0063] v=W3[concat(H′0,H′1,H′2)]+b3 (4)
[0064] Where v represents the sentence encoding vector, concat represents the concatenation of vectors, W3 and b3 are trainable parameters, W3 is a matrix, b3 is a vector, and W3∈R m×3m .
[0065] Compared with existing technologies, this invention employs a BERT entity-aware sentence encoding method. The pre-trained BERT language model effectively captures contextual information and learns semantic information within sentences. Using the pre-trained BERT language model as the word embedding module, the hidden state vectors obtained after BERT processing of the sentence header (CLS), header target entities, and tail target entities are linearly transformed and then concatenated. The resulting vector is then used as the sentence's vector representation. This processing method is simple and efficient. Furthermore, the introduction of header and tail target entities as supplementary components to the CLS label in the BERT pre-trained language model fully leverages the advantages of the BERT pre-trained language model.
[0066] The sentence packet representation vector encoding module 12 is used to execute step S12 to obtain the sentence packet representation vector 1 of sentences containing the same entity pair according to the sentence encoding vector 1.
[0067] Specifically, step S12 involves obtaining a sentence packet representation vector one, which is composed of sentences containing the same entity pair, based on the sentence encoding vector one.
[0068] In this step, sentences containing the same entity pair are grouped into a sentence packet, and the sentence packet is represented to obtain a sentence packet representation vector that expresses the corresponding entity pair and the relationship between the entity pair.
[0069] In a specific implementation, a sentence bag consisting of N sentences containing the same entity pair is represented as B = {s1, s2, ..., s...} i ,...,s N}, where s i This represents the i-th sentence in the sentence bag. The sentence bag representation vector corresponding to this sentence bag is as follows:
[0070]
[0071]
[0072] Where g represents the sentence package and vector one, g∈R m v i Sentence s i The sentence encoding vector one, a i This represents the sentence s obtained through the selective attention mechanism. i The weight of r represents the degree of matching between entity pairs in a sentence and their corresponding entity pair relations. A higher weight indicates a better match, and vice versa. j Let each sentence relation representation vector be a trainable sentence relation representation vector, and let each sentence relation representation vector correspond to a corresponding sentence relation. Furthermore, each sentence relation representation vector can be obtained from a sentence relation query matrix, where j∈(0,1). That is, the sentence relation query matrix in this step is composed of two sentence relation representation vectors, denoted as r0 and r1, where r0∈R. m , r1∈R m .
[0073] In this step of the scheme, for each sentence packet, the corresponding sentence packet representation vector one needs to be calculated for each of the sentence relationship representation vectors.
[0074] The relation representation value calculation module 13 is used to execute step S13 to obtain the relation representation value of the sentence packet representation vector 1 according to the sentence packet representation vector 1.
[0075] Specifically, S13 involves obtaining the relation representation value of the sentence packet representation vector one based on the sentence packet representation vector one.
[0076] In a specific implementation, the relation representation value is used to represent the degree to which the sentence package representation vector expresses the corresponding sentence relationship. In this case, the relation representation value is calculated in the following way:
[0077] P(r j |g i = softmax(W) c g i +b c (7)
[0078] Wherein, P(r) j |g i ) represents a sentence package representing a vector g i expression r j The value represented by the entity in the relation, W. c b c All are trainable parameters, W c ∈R l×m , l represents the total number of entity pairs in the sentence. In this embodiment, l = 2.
[0079] In this step of the scheme, for each sentence packet, the two sentence packet representation vectors need to be calculated separately. Therefore, for each sentence packet, two relation representation values can be obtained, which correspond to the sentence relation representation vectors r0 and r1 respectively, that is, they correspond to the corresponding sentence relation. The magnitude of their values indicates the degree to which the sentence expresses the corresponding sentence relation, and their values are within [0, 1] and the sum of the values of the two vector elements is 1.
[0080] The sentence relationship acquisition module 14 is used to execute step S14 to obtain the sentence relationship representation corresponding to the sentence packet based on the relationship representation value of the sentence packet representation vector.
[0081] Specifically, step S14 involves obtaining the sentence relation representation corresponding to the sentence packet based on the relation representation value of the sentence packet representation vector one.
[0082] For each sentence packet, the sentence relation representation ultimately expressed by the sentence in the corresponding sentence packet is determined by comparing the numerical values of its two relation representation values.
[0083] Specifically, in this embodiment, the sentence relation representations in this step are 0 and 1, respectively. 0 indicates that the sentence has no potential semantic value, and 1 indicates that the sentence has potential semantic value. The relation representation value corresponding to r0 represents the degree to which the corresponding sentence expresses the sentence relation represented by relation representation 0, and the relation representation value corresponding to r1 represents the degree to which the corresponding sentence expresses the sentence relation represented by relation representation 1. Therefore, the corresponding sentence relation representation can be determined by comparing the magnitudes of the relation representation values. Furthermore, the determined sentence relation representation is output. When the output is 0, it indicates that the sentence has no potential semantic value; when the output is 1, it indicates that the sentence has potential semantic value.
[0084] Furthermore, when determining the sentence relation representation, if the number of maximum relation representation values exceeds one, the sentence relation representation corresponding to the highest value is taken. Therefore, in the steps of this scheme, when both relation representation values are 0.5, it is determined that the sentence expresses the relation corresponding to 0.
[0085] In this scheme, since some of the data marked as NA after the remote supervision relationship is extracted does indeed express no relationship, while some data is incorrectly marked as NA due to data incompleteness, this step S14 needs to filter out the data that incorrectly marked as NA due to data incompleteness.
[0086] The FN noise dataset acquisition module 15 is used to perform step S15 to obtain an FN noise dataset formed by sentences with potential semantic value based on the sentence relation representation.
[0087] Specifically, step S15 involves obtaining an FN noise dataset composed of sentences with potential semantic value based on the sentence relation representation.
[0088] In a specific embodiment, for the corresponding sentence package, when the output result after processing in step S14 is 0, it indicates that the sentences in the sentence package do not have potential semantic value; when the output result after processing in step S14 is 1, it indicates that the sentence has potential semantic value. At this time, all sentences in the sentence package can be stored as FN noise to obtain the FN noise dataset.
[0089] Therefore, in this scheme, after executing steps S11-S15 through submodule 1, the noise data labeled as NA after the remote supervision relationship is extracted can be filtered to obtain FN noise data.
[0090] The sub-model 2 includes a sentence encoding vector encoding module 21, a sentence package representation vector encoding module 22, a relation representation value calculation module 23, and a sentence relation acquisition and annotation module. The sentence relation acquisition and annotation module includes a sentence relation acquisition sub-module 241 and a sentence relation annotation sub-module 242.
[0091] The sentence encoding vector encoding module 21 is used to perform step S21 to obtain the sentence encoding vector of the sentence in the FN noise dataset.
[0092] Specifically, S21 involves encoding the sentences in the FN noise dataset to obtain the sentence encoding vector 2 of the sentences in the FN noise dataset.
[0093] In this step, the sentence is also encoded using the BERT entity-aware sentence encoding method from step S11. Furthermore, since the encoding method in this step is essentially the same as in step S11, with the only difference being the parameter values in the formula, please refer to step S11 above for details; it will not be repeated here.
[0094] The sentence packet representation vector encoding module 22 is used to execute step S22 to obtain the sentence packet representation vector 2 of the FN noise dataset according to the sentence encoding vector 2.
[0095] Specifically, step S22 involves obtaining the sentence packet representation vector 2 of the FN noise dataset based on the sentence encoding vector 2.
[0096] In this step, sentences with the same entity pairs are also grouped into a sentence packet, the sentence packet is encoded, and the sentence packet representation vector two is obtained using the encoding method that is basically the same as in step S12. For details, please refer to the aforementioned step S12.
[0097] This step differs from step S12 in the following two ways when encoding the sentence packet: 1) The parameter values in the formula are different; 2) The sentence relation representation vector r j The number of relationships is the same as the total number of relationships in the relation classification and annotation of the FN noise dataset; in a specific embodiment, in this step, the number of sentence relation representation vectors included in the sentence relation query matrix is 54, namely r′0, r′1, ..., r′ 53 Each of these relationships corresponds to a sentence relationship in the relation annotation of the FN noise dataset.
[0098] The relation representation value calculation module 23 is used to execute step S23 to obtain the relation representation value of the second sentence packet representation vector according to the second sentence packet representation vector.
[0099] Specifically, step S23 involves obtaining the relational representation value of the second sentence packet representation vector based on the second sentence packet representation vector.
[0100] In this scheme, similarly, the relational representation value of the second sentence packet representation vector is obtained using a calculation formula similar to that in step S13, as follows:
[0101] P(r′ j |g′ i ) = softmax(W′ c g i +b′ c (8)
[0102] Wherein, P(r′) j |g′i ) represents a sentence package representing a vector g′ i Expression r′ j The value represented by the entity in the relation, W′ c b′ c All are trainable parameters, W′ c ∈R l′×m l′ represents the total number of relations for relation classification and annotation in the FN noise dataset. In a specific embodiment, l′ = 54.
[0103] For each sentence packet, in step S22, a second sentence packet representation vector has been obtained for each sentence relation representation vector. Then, in this step S23, the same number of relation representation values as the total number of relations in the relation classification and annotation of the FN noise dataset can be obtained. Each relation representation value corresponds one-to-one with the corresponding sentence relation representation vector, that is, it corresponds to the corresponding sentence relation. The magnitude of its value indicates the degree to which the sentence expresses the corresponding sentence relation, and its value is within [0, 1] and the sum of the values of all relation representation values is 1.
[0104] The second sentence relationship acquisition and annotation module is used to execute step S24 to acquire the entity pair relationship of the sentence and annotate the entity pair relationship of the sentence based on the relationship representation value of the second sentence packet representation vector.
[0105] In this scheme, since the relation representation values calculated in step S23 correspond to the relations of each sentence, the relation relations can be determined by the relation representation values.
[0106] The sentence relation acquisition and annotation module 24 includes a sentence relation acquisition submodule 241 and a sentence relation annotation submodule 242. The sentence relation acquisition submodule 241 executes step S241 to acquire the sentence relation representation corresponding to the sentence package based on the relation representation value of the sentence package representation vector 2. The sentence relation annotation submodule 242 executes step S242 to annotate the entity pair relations of the sentence based on the sentence relation representation corresponding to the sentence package.
[0107] Step S241 is: to obtain the sentence relation representation corresponding to the sentence package based on the relation representation value of the sentence package representation vector two.
[0108] Similar to step S14, in this step, the relation representation values obtained in step S23 are compared to obtain the relation representation value with the largest value, and the sentence relation representation corresponding to the sentence packet is determined as the sentence relation representation corresponding to the largest relation representation value. After the sentence relation representation is determined, the corresponding sentence relation can be obtained.
[0109] It should be noted that when determining the sentence relation representation, if the number of maximum values of the relation representation exceeds one, the sentence relation representation corresponding to the highest value in the order is taken.
[0110] Step S242 is as follows: Based on the sentence relation representation corresponding to the sentence package, the entity pair relations of the sentence are labeled.
[0111] In this scheme, based on the sentence relation representation, the corresponding sentences are labeled as entity pairs corresponding to the sentence relation representation, thereby completing the labeling of sentence entity pairs in the FN noise dataset and achieving the purpose of reducing noise.
[0112] Furthermore, in this scheme, in order to make the noise filtering result, i.e., the obtained FN noise dataset, more consistent with the real situation and improve the accuracy of noise filtering, this scheme also uses the training dataset to train and optimize the parameters in steps S11-S15 of sub-model 1. The specific optimization method is as follows:
[0113] S31. Initialize the network parameters in steps S11-S15.
[0114] S32. Perform steps S11-S15 on the positive example training dataset.
[0115] In this scheme, the positive example training dataset is a dataset in which the entity pair relations of sentences are not NA.
[0116] In a specific embodiment, the positive example training dataset uses the NYT-10 positive example dataset, in which the relationships between entities in sentences are non-NA.
[0117] S33. Calculate the total loss value after each iteration. Stop iterating when the total loss value tends to stabilize after multiple iterations.
[0118] In this scheme, each complete traversal of all samples in the positive training dataset is called an iteration.
[0119] In a specific embodiment, after each iteration, the loss value is calculated according to the following formula:
[0120]
[0121] Among them, L cls The cross-entropy loss value of the positive training dataset of sub-model 1 is represented by P, where P represents the positive training dataset. This indicates that the sentence packet representation vector of the positive example training dataset is processed.
[0122] Similarly, to improve the accuracy of sub-model 2 in reclassifying and labeling sentence entity pairs in the FN noise dataset, after sub-model 1 is optimized, this invention further optimizes the parameters of sub-model 2 using the training dataset. In this scheme, the optimization of the parameters in sub-model 2 employs generative adversarial learning and contrastive learning, as detailed below:
[0123] S41. Initialize the network parameters in steps S21-S23.
[0124] S42. Execute steps S11-S15 after parameter optimization using the negative training dataset to obtain the FN noise dataset of the negative training dataset.
[0125] In this scheme, the negative example training dataset is a dataset in which the entity pair relation of a sentence is NA.
[0126] In a specific embodiment, the negative example training dataset uses the NYT-10 negative example dataset, which is an NA-labeled dataset. This NA-labeled dataset includes two types of datasets: one type contains sentences that actually express NA relationships, i.e., there is no relationship between entities; the other type contains FN noise. By performing the parameter optimization steps S11-S15 on this negative example dataset, the data in the negative example dataset can be filtered to obtain the corresponding FN noise dataset.
[0127] S43. Generate adversarial learning based on the FN noise dataset obtained in step S42 and the positive example training dataset.
[0128] In this step, generative adversarial learning uses a generator and a discriminator, wherein the sentence packet representation vector encoding module 22 in sub-model 22 serves as the generator in this adversarial learning process. The specific steps are as follows:
[0129] Step S21 is performed on both the FN noise dataset obtained in step S42 and the positive example training dataset. Then, using the generator, step S22 is performed to obtain the FN noise data sentence packet representation vector and the positive example training data sentence packet representation vector, respectively. Specifically, the FN noise data sentence packet representation vector is the sentence packet representation vector corresponding to the FN noise dataset, and the positive example training data sentence packet representation vector is the sentence packet representation vector corresponding to the positive example training dataset.
[0130] A discriminator is used to discriminate the data generated by the generator.
[0131] In a specific embodiment, the FN noise dataset is the FN noise dataset obtained after performing parameter optimization steps S11-S15 on the negative example dataset of NYT-10, and the positive example training dataset uses the positive example dataset of NYT-10. Furthermore, for each sentence packet representation vector generated by the generator, the discriminator performs the following calculations:
[0132] o i ′=σ(W d g i ′+b d (10)
[0133] Among them, o i ′ represents a sentence package, and g represents a vector. i The probability distribution corresponding to ′, g i ′∈R m σ() represents the sigmoid function, W d b d This represents the trainable parameters.
[0134] To achieve adversarial training, the parameters are optimized by calculating an adversarial loss, where the generator's loss is expressed as:
[0135] L g =-E s~M [logD(G(s))] (11)
[0136] Among them, L g Let E represent the generator loss value, M represent the FN noisy dataset, s represent the sentence in the FN noisy dataset M, and E represent the generator loss value. s~M [] indicates processing the sentence encoding vector of the sentence in the FN noise dataset M, G(s) indicates inputting the sentence encoding vector of sentence s into the sentence packet representation vector generated by the generator, and D(G(s)) indicates inputting the sentence packet representation vector generated by the generator into the probability distribution o' obtained by the discriminator.
[0137] The loss of the discriminator can be expressed as:
[0138]
[0139] Among them, L d Let P represent the discriminator loss value, P represent the positive training dataset, and s represent the discriminator loss value. i Let D(s) represent sentences in the positive example training dataset. i ) represents the probability distribution o' obtained by inputting the sentence packet representation vectors of sentences in the positive example training dataset into the discriminator. This indicates that the sentence packet representation vector of the positive example training dataset is processed, s jThis represents a sentence in the FN noise dataset M. This indicates that the sentence encoding vectors of sentences in the FN noise dataset M are processed.
[0140] S44. Calculate the contrastive loss based on the sentence packet representation vector of the positive example training data.
[0141] Since different entity pairs can express the same relationship, this scheme uses the sentence packet representation vector of the positive example training dataset to cluster data with the same relationship. Specifically, this scheme calculates the contrastive loss by increasing the distance between samples with different relationships and decreasing the distance between samples with the same relationship.
[0142] In a specific embodiment, for a positive example training dataset, given a sentence packet representation vector of one of the positive example training data, all other data in the positive example training dataset are divided according to whether they express the same relationship as the sentence packet representation vector of the positive example training data or express a different relationship, and the following contrastive loss representation method is adopted:
[0143]
[0144] dist + (g′,g′ i )=(max(τ-cos(g′,g′ i ),0)) 2 (14)
[0145] dist_(g′,g′ j )=1-(max(cos(g′,g′ j ),0)) 2 (15)
[0146] Among them, L ctra The contrarian loss value is represented by τ, which is a hyperparameter preset before training to avoid packet representation collapse, and cos() represents the cosine function. Q + Q represents the set of sentence bag representation vectors that express the same relation as the given positive example training data sentence bag representation vectors. _ Dist represents a set of sentence packet representation vectors that express different relations from the given positive example training data. + (g′,g′ i The expression `dist_(g′,g′)` represents operations on the sentence packet representation vectors with the same relation. j ) indicates operations performed on the sentence package representation vectors of different relations.
[0147] For the contrastive loss function described above, the closer the distance between two data samples, the smaller the angle between them in the coordinate system, and the closer the value of the cosine function is to 1, τ-cos(g′,g′). i The smaller the value, the smaller the dist will be. + (g′,g′ i The smaller the value of dist_(g′,g′), the larger the angle in the coordinate system, and the smaller the value of the cosine function. j The larger the value of L, the closer it is to 1. Furthermore, when a decreasing value is subtracted from a increasing value, the overall value continuously decreases. ctra When the minimum value is reached, dist + (g′,g′ i The minimum value is taken, which represents the sample with the same relation that is closest in distance, dist_(g′,g′). j The maximum value is found when the distance between different relationships is greatest. Therefore, gradient descent can be used to find L. ctra The extreme value.
[0148] S45. Calculate the relation representation value based on the FN noise data sentence packet representation vector and the positive example training data sentence packet representation vector, and further calculate the classification loss value.
[0149] In this scheme, the value of the relation representation is calculated using the following formula:
[0150] P(r′ j |g′ i ) = softmax(W′ c g′ o +b′ c (16)
[0151] Among them, W′ c b′ c All are trainable parameters, W′ c ∈R l′×m , g′ i ∈R m And l′ represents the total number of sentence relations. In the specific implementation, l′ = the number of relations in the NYT10 dataset, which is 54 in total.
[0152] Furthermore, the classification loss value of the positive training dataset P is calculated.
[0153]
[0154] Among them, L cls This represents the cross-entropy classification loss value.
[0155] S46. After each iteration, calculate the total loss value of sub-model 2, and perform the steps of sub-model 1 and sub-model 2 on the test set samples. At the same time, perform performance testing on sub-model 2, and repeat multiple iterations until the performance no longer improves.
[0156] In this step, the total loss value is calculated using the following formula:
[0157] L = L g + L d + L ctra + L cls (18)
[0158] Where L represents the total loss value of the second sub-model.
[0159] In this scheme, each complete traversal of all samples in the training set is called an iteration. After each iteration, using samples from the test set, the steps of sub-model 1 and sub-model 2 are executed to test the performance of sub-model 2, and the current performance test results and corresponding network parameters are recorded. If the performance no longer improves after multiple iterations, it indicates that sub-model 2 has been trained to a convergent state (i.e., the optimal performance state). The following metrics were used for performance testing: AUC, P@N, PR-curve, and Micro-F1.
[0160] During multiple iterations of the training dataset for sub-model 2, the total loss value of the training dataset corresponding to sub-model 2 is controlled by gradient descent, and the parameters in sub-model 2 are adjusted and optimized accordingly. This process is repeated until the performance no longer improves, at which point it stops.
[0161] In this invention, for noisy data obtained after remotely supervised relation extraction, sentence information is extracted quickly and accurately using BERT entity-aware sentence encoding, thereby improving subsequent noise filtering and classification performance. Furthermore, by incorporating contrastive learning and generative adversarial learning into the training process, noise and real data are placed in a unified distribution, ultimately achieving better relation classification results and thus improving noise reduction.
[0162] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and the present invention also intends to include these modifications and variations.
Claims
1. A noise reduction method for remote supervised relation extraction based on BERT entity perception, characterized in that, Includes the following steps: S11. The sentence is encoded using a BERT entity-aware sentence encoding method to obtain a sentence encoding vector containing entity pair information. The BERT entity-aware sentence encoding method includes: using a pre-trained language model BERT to embed words into the sentence, and obtaining the output vector representations of the head target entity, the tail target entity, and the sentence head marker based on the hidden state vectors of the head target entity, the tail target entity, and the sentence head marker obtained by BERT; and then obtaining sentence encoding vector one based on the output vector representations of the head target entity, the tail target entity, and the sentence head marker. Furthermore, the sentence is noisy data labeled NA obtained after extraction using a remote supervision relation; S12. Based on the sentence encoding vector one, obtain the sentence packet representation vector one of the sentence packets composed of sentences containing the same entity pair; S13. Obtain the relation representation value of the sentence packet representation vector one according to the sentence packet representation vector one; S15. Based on the sentence relation representation of the sentence package obtained from the relation representation value, obtain the FN noise dataset formed by sentences with potential semantic value; S2. Classify the sentence relationships in the FN noise dataset.
2. The noise reduction method according to claim 1, characterized in that: In step S11, the sentence encoding vector is represented as follows: v=W3[concat(H′0,H′1,H′2)]+b3 Where v represents the sentence encoding vector, concat represents the concatenation of vectors, H0′ represents the output vector representation of the sentence header marker, H1′ represents the output vector representation of the header target entity, H2′ represents the output vector representation of the tail target entity, and W3 and b3 are both trainable parameters.
3. The noise reduction method according to claim 2, characterized in that: The output vector representations of the head and tail target entities are obtained using the following formula: wherein W1, W2, b1, b2 are trainable parameters, and H i to H j denotes the hidden state vector of the head target entity obtained by BERT, H k to H m denotes the hidden state vector of the tail target entity obtained by BERT, and tanh is a hyperbolic tangent activation function.
4. The noise reduction method according to claim 3, characterized in that: The output vector representation of the sentence header marker is obtained using the following formula: H′0=W0[tanh(H0)]+b0 Here, H0 represents the hidden state vector of the sentence header obtained by BERT, and W0 and b0 are trainable parameters.
5. The noise reduction method according to claim 4, characterized in that: In step S12, the sentence packet representation vector is represented as follows: Where g represents the sentence package, v represents vector one, and v represents vector one. i Let a represent the sentence encoding vector of the i-th sentence in the sentence packet. i This represents the weight of the i-th sentence obtained through the selective attention mechanism.
6. The noise reduction method according to claim 5, characterized in that: The formula for calculating the weight is as follows: Where N represents the total number of sentences in the sentence package, r j The sentence relation is represented by a vector, j∈(0,1).
7. The noise reduction method according to claim 6, characterized in that: In step S13, the formula for calculating the relation representation value is as follows: P(r j |g i )=softmax(W c g i +b c ) Wherein, P(r) j |g i ) represents a sentence package representing a vector g i expression r j The value represented by the entity in the relation, W. c b c All of these are trainable parameters.
8. The noise reduction method according to any one of claims 1-7, characterized in that: Step S2 includes: S21. The sentences in the FN noise dataset are encoded using a BERT entity-aware sentence encoding method to obtain the sentence encoding vector 2 of the sentences in the FN noise dataset. S22. Based on the second sentence encoding vector, obtain the second sentence packet representation vector of the sentence packet composed of sentences containing the same entity pair in the FN noise dataset; S23. Obtain the relation representation value of the second sentence packet representation vector based on the second sentence packet representation vector; S24. Based on the relation representation value of the sentence package representation vector two, obtain the entity pair relation of the sentence and label the entity pair relation of the sentence.
9. A denoising model for remote supervised relation extraction based on BERT entity perception, comprising sub-model one and sub-model two, wherein, The first sub-model is used to obtain the FN noise dataset from the noise data labeled as NA after the remote supervision relation is extracted, and the second sub-model is used to classify the sentence relations of sentences in the FN noise dataset. The sub-model one includes a sentence encoding vector encoding module one, a sentence packet representation vector encoding module one, a relation representation value calculation module one, and an FN noise dataset acquisition module; The sentence encoding vector encoding module one uses a BERT entity-aware sentence encoding method to encode the sentence, resulting in a sentence encoding vector one containing entity pair information; The sentence packet representation vector encoding module one is used to obtain the sentence packet representation vector one of sentences containing the same entity pair based on the sentence encoding vector one; The relation representation value calculation module one is used to obtain the relation representation value of the sentence packet representation vector one based on the sentence packet representation vector one; The FN noise dataset acquisition module is used to obtain an FN noise dataset composed of sentences with potential semantic value based on the sentence relation representation of the sentence package obtained from the relation representation value.
10. The noise reduction model according to claim 9, characterized in that: The second sub-model includes a sentence encoding vector encoding module two, a sentence package representation vector encoding module two, a relation representation value calculation module two, and a sentence relation acquisition and annotation module, wherein: The second sentence encoding vector encoding module uses the BERT entity-aware sentence encoding method to encode the sentence and obtain the second sentence encoding vector of the sentence in the FN noise dataset; The second sentence packet representation vector encoding module is used to obtain the second sentence packet representation vector of the sentence packet composed of sentences containing the same entity pair in the FN noise dataset based on the second sentence encoding vector; The second relation representation value calculation module is used to obtain the relation representation value of the second sentence package representation vector based on the second sentence package representation vector; The sentence relationship acquisition and annotation module is used to obtain the entity pair relationship expressed by the sentence and annotate the entity pair relationship of the sentence based on the relationship representation value of the sentence package representation vector two.
Citation Information
Patent Citations
Entity relation extraction method
CN108733792A
Document level relation extraction method and device, electronic equipment and storage medium
CN115358220A
