A Method for Denoising Remote Supervision Datasets

Abstract

The present invention discloses a method for denoising a remote supervision data set. First, the DS data set is divided into a positive example set TD and a negative example set FD according to whether the data has a label. Then, a high-quality positive example set CTD is extracted from TD through a pattern-based data extraction algorithm. Using CTD and FD, a binary classification model Filter-Net is trained, and this model can accurately identify a high-quality negative example set CFD from FD. Finally, the high-quality negative example set CFD and the positive example set TD are used together as the training data set of the denoising model RL to obtain a higher-quality correctly labeled sample set. Starting from making full use of the mislabeled data in the DS data set, the present invention proposes a method for extracting high-quality positive and negative example data from the DS data set, and uses the high-quality positive and negative example data for the training of the denoising model, thereby improving the accuracy of the denoising model and achieving the purpose of effective denoising.

Description

Technical Field

[0001] The present invention relates to network data processing, and in particular to a method for denoising a remotely supervised data set. Background Art

[0002] Relation classification is a basic task in information extraction and an important part of the knowledge graph construction process. [Robust distant supervision relation extraction via deep reinforcement learning. Qin, P.; Xu, W.; and Wang, W.Y. 2018. In ACL, 2137–2147] pointed out that relation classification is to determine which relation two entities in a given sentence belong to. Relation classification essentially belongs to a multi-classification problem. With the rise of neural networks, more and more researchers use neural networks to solve this problem. BILSTM proposed in 2015 and BILSTM+ATT proposed in 2016 have achieved remarkable results in relation extraction. However, using neural networks to solve the relation classification problem requires a large amount of labeled data to ensure effective training of the model. However, obtaining a large amount of labeled data requires a great deal of cost. To reduce the cost of obtaining a large amount of labeled data, distant supervision (DS) was proposed in 2009, which automatically annotates data by aligning a knowledge base with unstructured text. Distant supervision is based on the assumption that for a triple (consisting of a pair of entities and a relation) in a given knowledge graph, any sentence in the external document library that contains this pair of entities can be defined as this relation, otherwise the relation is labeled as 'none'. The weak assumption of DS for defining entity relations does not conform to the actual situation. Therefore, DS cannot obtain a data set with completely correct entity relation annotations, and there may be a situation where 30% of the data in the data set obtained by distant supervision is wrongly labeled data.

[0003] In the literature [Reinforcement learning for relation clas - sification from noisy data. Feng, J.; Huang, M.; Zhao, L.; Yang, Y.; and Zhu, X. 2018. In Thirty - Second AAAI Conference on Artificial Intelligence], the authors proposed an RL model for extracting the positive example dataset of DS. RL accepts the labeled data in DS, discriminates whether the labels are correctly annotated, and removes some mis - annotated data to achieve the effect of denoising the labeled data. RL is essentially a binary - classification model. To be able to identify correctly annotated data, the training data of the model must contain both correctly annotated and mis - annotated examples. This paper believes that simply using the labeled and unlabeled data in the DS dataset as positive and negative examples to train the model is insufficient because among the positive examples, there are some mis - annotated data, and similarly, among the negative examples, there are also some data with obvious relationships that are not correctly annotated. This phenomenon is the key reason why the RL model cannot well extract the correctly annotated data in the DS dataset. Summary of the Invention

[0004] In view of the above - mentioned deficiencies in the prior art, the present invention provides a method for denoising a remotely supervised dataset.

[0005] To achieve the above - mentioned invention objective, the technical solution adopted by the present invention is as follows:

[0006] A method for denoising a remotely supervised dataset includes the following steps:

[0007] S1. Obtain a remotely supervised dataset and divide the obtained data into a positive sample set and a false negative sample set according to whether there are labels;

[0008] S2. Use a pattern - based data extraction method to extract the data in the positive sample set obtained in step S1 to obtain high - quality positive sample data;

[0009] S3. Use the high - quality positive sample data extracted in step S2 and the data in the false negative sample set obtained in step S1 as the training data of a negative sample screening network for training to obtain a high - quality negative sample set;

[0010] S4. Use the high - quality negative sample set identified in step S3 and the data in the positive sample set obtained in step S1 as the training dataset of a denoising model and perform denoising model training to obtain correctly annotated positive sample data.

[0011] The beneficial effect of the above solution is that it fully considers the role of the features of negative samples in DS in the learning and extraction of positive samples, and further improves the effect of denoising the DS dataset.

[0012] Further, the specific steps of step S2 include:

[0013] S21. Arbitrarily select the pattern word m included between two entity sentences in the positive sample set i ;

[0014] S22. Determine whether the word m selected in step S21 i is in the pattern. If not, incorporate it into the pattern set M. If so, increment the count of the pattern set M by 1;

[0015] S23. Repeat steps S21 - S22 until all sentences are processed. Determine whether the count of the pattern words in the pattern set M is greater than the set threshold. If so, the selected pattern word m i is a high - frequency pattern, and the corresponding selected positive sample data is high - quality positive sample data.

[0016] The beneficial effect of the above further solution is to screen out more valuable positive sample data in the positive sample set as the training data for the subsequent model.

[0017] Further, the high - quality positive sample data in step S22 is expressed as:

[0018]

[0019] where CTDS is high - quality positive sample data, s i is the remotely supervised data obtained in step S1, t j is the label, and r is the threshold parameter;

[0020] represents the probability of the performance pattern pattern of the i - th positive sample data, p i is for pattern i is the number of times pattern appears in the entire remotely supervised dataset, and n is the total number of patterns.

[0021] Further, the specific method of step S3 is as follows:

[0022] S31. Use the high - quality positive sample dataset obtained in step S2 and the false negative sample set in step S1 as the training set to construct a CNN convolutional neural network;

[0023] S32. Calculate the loss of the CNN convolutional neural network constructed in step S31;

[0024] S33. Adjust the parameters of the CNN network until the network loss is less than the set threshold, and output high-quality negative sample data.

[0025] The beneficial effect of the above further solution is that the data that cannot be recognized by distant supervision and is assigned to the negative sample set is screened out by the false negative example screening network.

[0026] Further, in step S32, the FocalLoss function is used to calculate the loss of the CNN convolutional neural network, which is expressed as:

[0027] FL(pt)=-(1 - P t ) γ logP t ;

[0028] where P t is the classification probability that the data x is labeled as the negative sample label y, and γ is the scaling factor.

[0029] The beneficial effect of the above further solution is that the trained CNN convolutional neural network is converged by using the loss function.

[0030] Further, the high-quality negative sample data in step S33 is expressed as:

[0031]

[0032] where CFD is the high-quality negative sample data, p represents the probability that the predicted sample belongs to 1 (the range is 0 - 1), y represents the negative sample label, and the value range of y is {+1, -1}. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic flow chart of a method for denoising a distant supervision data set according to the present invention.

[0034] Figure 2 It is a flow chart of the high-quality positive sample data extraction algorithm SelePD based on patterns according to an embodiment of the present invention.

[0035] Figure 3 It is a schematic comparison curve diagram of the PR curves of PRL_data and OLD in the CNN + ATT model according to an embodiment of the present invention.

[0036] Figure 4 It is a schematic comparison curve diagram of the PR curves of PRL_data and OLD in the PCNN + ONE model according to an embodiment of the present invention.

[0037] Figure 5 It is a schematic comparison curve diagram of the PR curves of PRL_data and OLD in the PCNN + ATT model according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0038] The specific embodiments of the present invention will be described below to facilitate those skilled in the art to understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention defined and determined by the appended claims, these changes are obvious, and all inventions made using the concept of the present invention are within the scope of protection.

[0039] A method for denoising a remotely supervised dataset, as Figure 1 shown, includes the following steps:

[0040] S1. Obtain a remotely supervised dataset and divide the obtained data into a positive sample set and a false negative sample set according to whether there is a label;

[0041] Here, the label refers to the annotation of the entity relationship in the dataset. For example, if there is a relationship between two entities "A" and "B" in the dataset, and the relationship is "C", then the label of the pair of entities (A, B) is "C". If there is no relationship between the entity pairs, the label is "none".

[0042] S2. Use a pattern-based data extraction method to extract the data in the positive sample set obtained in step S1 to obtain high-quality positive sample data;

[0043] Specifically, step S2 specifically includes:

[0044] S21. Arbitrarily select a pattern word m included between two entity sentences in the positive sample set i ;

[0045] S22. Determine whether the word m selected in step S21 i is in the pattern set M. If not, incorporate it into the pattern set M. If so, increment the count of the pattern set M by 1;

[0046] S23. Repeat steps S21 - S22 until all sentences are processed. Determine whether the count of the pattern words in the pattern set M is greater than a set threshold. If so, the selected pattern word m i is a high-frequency pattern, and the corresponding selected positive sample data is high-quality positive sample data.

[0047] S3. Use the high-quality positive sample data extracted in step S2 and the data in the false negative sample set obtained in step S1 together as the training data for the negative sample screening network for training;

[0048] Specifically, the specific manner of step S3 is:

[0049] S31. Construct a CNN convolutional neural network using the high-quality positive sample dataset and the false negative sample set obtained in step S2 as the training set;

[0050] S32. Calculate the loss of the CNN convolutional neural network constructed in step S31, expressed as:

[0051] FL(pt) = -(1 - P t ) γ logP t ;

[0052] where Pt is the classification probability that the data x is labeled as the label y, and γ is the scaling factor.

[0053] The principle of the FocalLoss function for screening negative samples is as follows:

[0054] FocalLoss adds a modulation factor 1 - P to the cross-entropy. t When classifying negative examples, since negative examples are easy to learn, the value of Pt will be very large. At this time, the value of pt tends to 1, so the modulation factor 1 - P t tends to 0, and the loss of negative examples will increase. Therefore, negative examples are suppressed when classified. When the model classifies positive examples, since the number of positive samples is small and not easy to learn, the value of Pt will be very small, and the value of Pt tends to 0. Then 1 - P t tends to 1, and the overall loss remains unchanged. At this time, positive examples are not affected when classified, so they can be well learned. Simply put, the FocalLoss modulation factor 1 - P t suppresses the class with a large number, so that the class with a small number can also be fully learned, where γ is the scaling factor.

[0055] S33. Adjust the CNN network parameters until the network loss is less than the set threshold, and output the high-quality negative sample data.

[0056] S4. Use the high-quality negative sample set identified in step S3 and the positive sample set data obtained in step S1 together as the training dataset of the denoising model and perform denoising model training to obtain correctly labeled positive sample data.

[0057] For the convenience of verification, this paper uses several models that perform outstandingly in relation classification on the NYT dataset for verification, including the CNN+ATT, PCNN+ONE, and PCNN+ATT models. Use our model to train the noisy dataset, and judge whether our model has an improvement effect on the performance of the model through the PR curve and AUC index of the model.

[0058] The processed training dataset PRL_data and the original data OLD without noise removal in this paper were used to train the CNN+ATT, PCNN+ONE, and PCNN+ATT models respectively. From Figure 3 , Figure 4 , Figure 5 's results show that the data denoised by the model in this paper is more helpful for improving the performance of the model. The denoising model proposed in this paper has a more obvious improvement in the performance of the PCNN+ONE and PCNN+ATT models. Especially for the PCNN+ATT model, the improvement effect is more significant, and the PR curve performs better.

[0059] In the same way as the comparison method of the PR curve. We used the data PRL_data and OLD to train each model respectively. From the results in Table 1, compared with the original data OLD, the data denoised by the model in this paper is more helpful for improving the performance of the model. The model in this paper has a particularly obvious improvement in the performance of the PCNN+ONE and PCNN+ATT models. Especially for the PCNN+ATT model, the improvement effect is more significant. The datasets obtained by our improved method perform better in each model than the unprocessed datasets.

[0060] Table 1 Comparison of AUC values of models

[0061] Model OLD PRL_data CNN+ATT 0.204 0.220 PCNN+ONE 0.212 0.228 PCNN+ATT 0.218 0.235

[0062] This invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the invention. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for realizing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0063] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including instruction means, and the instruction means realizes the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0064] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so as to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing steps for implementing the functions specified in one flow Figure 1 one flow or more flows and / or boxes Figure 1 or steps for implementing the functions specified in a box or more boxes.

[0065] In the present invention, specific embodiments are used to illustrate the principles and implementation manners of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation on the present invention.

[0066] Those of ordinary skill in the art will realize that the embodiments described herein are for helping the reader understand the principles of the present invention, and it should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those of ordinary skill in the art can make various other specific deformations and combinations without departing from the essence of the present invention based on the technical revelations disclosed in the present invention, and these deformations and combinations are still within the protection scope of the present invention.

Claims

1. A method for denoising a remotely supervised dataset, characterized in that, It includes the following steps: S1. Obtain a remotely supervised dataset and divide the obtained data into a positive sample set and a false negative sample set according to whether there is a label. Specifically, if there is a relationship between the two entity sentences "A" and "B" in the dataset and the relationship is "C", then the label of the pair of entity sentences (A, B) is "C"; if there is no relationship between the entity sentence pairs, the label is "none". S2. Use a pattern-based data extraction method to extract the data in the positive sample set obtained in step S1 to obtain high-quality positive sample data. Specifically, it includes the following steps: S21. Arbitrarily select the pattern words included between two entity sentences in the positive sample set ; S22. Determine the word selected in step S21 whether it is in the pattern. If not, incorporate it into the pattern set M. If so, increment the count of the pattern set M by 1; S23. Repeat steps S21 - S22 until all sentences are processed. Determine whether the count of the pattern words in the pattern set M is greater than the set threshold. If so, the selected pattern words are high-frequency patterns, and the corresponding selected positive sample data are high-quality positive sample data; S3. Use the high-quality positive sample dataset extracted in step S2 and the data in the false negative sample set obtained in step S1 together as the training data of a high-quality negative sample screening network for training to obtain a high-quality negative sample set. S4. Use the high-quality negative sample set identified in step S3 and the positive sample set data obtained in step S1 together as the training dataset of a denoising model and perform denoising model training to obtain correctly labeled positive sample data.

2. The remote supervision dataset denoising method according to claim 1, wherein, The high-quality positive sample data in step S22 is represented as: Among them, is high-quality positive sample data, is the i-th remotely supervised data obtained in step S1, n is the quantity of, is the j-th label, m is the quantity of, is the threshold parameter; Indicates the i pattern of the positive sample data, is the number of occurrences in the entire remotely supervised dataset, and n is the total number.

3. The remote supervision dataset denoising method according to claim 2, characterized in that, The specific method of step S3 is: S31. Combine the high-quality positive sample data obtained in step S2 and the negative sample set in step S1 as the training dataset. S32. Construct a CNN convolutional neural network according to the training dataset in step S31; calculate the loss of the CNN convolutional neural network constructed in step S31. S33. Adjust the CNN network parameters until the network loss is less than a set threshold, and output high-quality negative sample data.

4. A remote supervision dataset denoising method according to claim 3, characterized in that In the step S32, the FocalLoss function is used to calculate the loss of the CNN convolutional neural network, which is expressed as: ; Among them, among them is data labeled as a negative sample label is the classification probability of, is the scaling factor.

5. A method for denoising a remote supervision dataset according to claim 4, characterized in that, The high-quality negative sample data in step S33 is represented as: Among them, is high-quality negative sample data, represents the probability that the predicted sample belongs to 1 (the range is 0-1), represents the negative sample label, and its value range is {+1, -1}.

Images