Knowledge graph-based weakly supervised data labeling rule discovery method and system

Abstract

The application discloses a kind of weak supervision data labeling rule discovery method and system based on knowledge graph, to reduce the rule of error classification, design a kind of label-aware rule generation method, priori knowledge is propagated from seed rule to candidate rule based on rule knowledge graph.For reduce the sentence of error matching, a kind of semantic matching mechanism based on cross attention is proposed, based on rule knowledge graph, while refining sentence semantic information, enrich rule semantic information.To improve the overall quality of rule discovery, an inconsistency-oriented active learning strategy is proposed to verify the inconsistent rules in rule generation and matching.The application can solve the problem of low precision and low recall caused by semantic drift in rule discovery, and improve the quality of data labeling under limited prior knowledge.

Description

Technical Field

[0001] This invention belongs to the field of data annotation technology, and relates to a method and system for discovering weakly supervised data annotation rules, specifically a method and system for discovering weakly supervised data annotation rules based on knowledge graphs. Background Technology

[0002] Data labeling is a task that involves adding corresponding labels to text based on the semantic information contained in the text corpus. In weakly supervised environments, deep learning models are prone to overfitting when the number of labeled samples is small. Rules, on the other hand, due to their good interpretability, maintainability, and minimal supervision, are helpful in extracting domain knowledge and achieving automated data labeling, which is of great significance to fields such as data mining and information retrieval. However, mismatched sentences and misclassified rules can lead to semantic drift in rule discovery, causing matched sentences to drift from the target label to other labels.

[0003] Existing rule discovery work can be divided into two categories: (1) using experts to formulate rules. For example, Snorkel uses experts to formulate rules and aggregates rule annotation results through probabilistic models. However, expert annotation is costly and has limited domain knowledge, making it difficult to cover the entire corpus. (2) automatically mining rules through seed data. For example, Darwin uses annotated sentences to generate rule trees and generates rules by traversing the rule trees.

[0004] Existing rule discovery methods consider rules in isolation, neglecting the semantic relationships between them. Semantic relationships between rules can provide semantic constraints for rule generation, enrich the semantic information of rules, and optimize rule matching. Summary of the Invention

[0005] To address the issue of semantic drift in rule discovery caused by insufficient labeled data in weakly supervised environments, this invention provides a knowledge graph-based method and system for weakly supervised data labeling rule discovery. This method can accurately identify positive and negative examples from unlabeled sentences, thereby improving the accuracy and recall of entity relation extraction rules.

[0006] The technical solution adopted by the method of the present invention is: a weakly supervised data annotation rule discovery method based on knowledge graph, comprising the following steps:

[0007] Step 1: Select a certain amount of data from the corpus for annotation, and divide the corpus into an annotated dataset S. seed and unlabeled dataset C;

[0008] Step 2: By mining the labeled dataset S seedThe longest common subsequence of the unlabeled dataset C is used to construct a seed rule set R; the longest common subsequence of the unlabeled dataset C is used to construct a candidate rule set R. can ;

[0009] Step 3: In the seed rule set R and candidate rule set R can In the process, we mine the multidimensional semantic relationships between rules and construct a rule knowledge graph G;

[0010] Step 4: In the rule knowledge graph G, infer the candidate rule set R based on the semantic relationships between rules. can The rule in the tag L mod (r i );

[0011] Step 5: In the rule knowledge graph G, the rules are converted into regular expressions to achieve pattern matching of the rules against the corpus;

[0012] Step 6: In the rule knowledge graph G described in step 3, the semantic matching of rules to the corpus is achieved through a semantic matching mechanism based on cross-attention;

[0013] Step 7: In the pattern matching and semantic matching results described in steps 5 and 6, select the tag with the most matches as the sentence tag to obtain the matching sentence set S. t Select the sentence tag with the most matching occurrences as the rule tag L. srn (r i );

[0014] Step 8: In the rule knowledge graph described in Step 4 and the matching sentence set described in Step 7, rules whose rule generation and rule matching performance are consistent are regarded as confidence rules, i.e., L mod (r i ) = L sen (r i Using an inconsistency-oriented active learning strategy, rules that are inconsistent in rule generation and rule matching are selected for manual verification, and the confidence rules and manual verification results are added to the rule set R.

[0015] Step 9: Repeat steps 3-8. When the manual annotation cost is exhausted, stop the iteration and output the rule set R and the matching sentence set S. t .

[0016] Preferably, in step 1, the data labeling rule is defined by the rule subject r. i The mapping formed by .b ​​and rule label l: r i .b→l; where the rule subject r i.b is a text sequence, represented as p = [w, sub-type, w, obj-type, w], where sub-type and obj-type represent the entity types of the subject and object, respectively, and w represents the context word sequence of the text containing the subject and object entities.

[0017] Preferably, in step 2, the mining of the longest common subsequence involves comparing multiple corpus sequences after removing stop words and punctuation marks from the corpus to find the longest common subsequence containing the subject and object, which serves as the rule. The mining of the longest common subsequence continues based on the mining results until no new sequence can be found.

[0018] Preferably, in step 3, the rule knowledge graph G = {(n i ,p m ,n j )|n i ,n j ∈V,p m ∈P}; where V=V L ∪V R.b V is a set of nodes. L and V R.b Let P represent the set of rule body nodes and the set of rule label nodes, respectively, where P = P B-L ∪P B-B ∪P L-L Let P be the set of edges. B-L ={(n i ,p m ,n j )|n i ∈V R.b ,n j ∈V L} represents the semantic relationship between the rule body and the rule tag, P B-B ={(n i ,p m ,n j )|n i ,n j ∈V R.b} represents the semantic relationship between rule subjects, P L-L ={(n i ,p m ,n j )|n i ,n j ∈V L} represents the semantic relationship between rule tags, n i ,n j Let p represent two nodes in the node set V. m For two nodes n i ,n jjThe edge between;

[0019] The multidimensional semantic associations between the rules refer to "belonging to," "implication," and "opposition" associations; the semantic association between the rule subject and the rule label is a "belonging to" association, indicating that the rule subject and the rule label contain the same semantic information; the associations between rule subjects and between rule labels include "implication" and "opposition" associations; an "implication" association indicates that there is the same semantic information between the premise and the hypothesis, and the hypothesis is derived from the premise, that is... When v i Contains v j Then v i True v j True; the "opposite" relation indicates that there is contradictory semantic information between the premise and the hypothesis, and the hypothesis cannot be derived from the premise, that is... When v i Contains v j Then v i True v j False; adjust and expand the semantic association of the rules according to the data annotation requirements.

[0020] Preferably, in step 4, a label-aware rule generation mechanism is used to infer the candidate rule set R based on the relationships between rules. can The rule in the tag L mod (r i );

[0021] The label-aware rule generation mechanism includes initial embedding, neighbor information propagation, and public information propagation;

[0022] The initial embedding is achieved by using the pre-trained language model BERT to generate vector representations of rule labels and rule bodies. and in The vector representing the label of the m-th rule; The vector representing the rule body of the i-th rule;

[0023] The neighbor information propagation is used to propagate semantic information between rule subjects or rule tags; among rule subjects, given a specific target rule subject r... i .b,N i ={(r i .b,p m ,r n .b)|(r i .b,p m ,r n b)∈G} contains r iAll neighboring rule subjects of .b; then the semantic information of these neighboring rule subjects can be aggregated in the following way:

[0024]

[0025]

[0026]

[0027] Where, θ(r) i .b,p m ,r n b) represents the trainable weight parameters, indicating the neighbor rule subject r. n .b and the target rule subject r i The relationship between .b and p m The weights they represent; For relation p m ; It is a trainable weight matrix, d and d ′ Let be the dimension of the matrix; Then it represents the set of neighbor rule subjects N. i The vector; when θ(r) i .b,p m ,r n The larger the value of .b), the stronger the individual neighbor rule subject vector. In calculating the vector of the neighbor rule subject set The greater the contribution, the better;

[0028] Integrate the semantic information contained in the neighboring rule body into the target rule body, and update the vector representation of the target rule body:

[0029]

[0030] Where f() is the activation function, W r The weight matrix is ​​trainable; similarly, the vector representation of the target rule label is updated by propagating neighbor information in the rule labels.

[0031] The aforementioned public information dissemination occurs between the rule subject and the rule tag, integrating the semantic information of the rule subject and the rule tag, and incorporating the rule subject information into the rule tag:

[0032]

[0033]

[0034] Where σ(k) m ,r ib) is a trainable weight parameter used to represent how much public information is passed from the rule subject to the rule label; B, W l and W c For trainable model parameters; l m This represents the m-th rule label; Indicates that it belongs to the rule label l m The set of rules;

[0035] Integrate the semantic information of the rule tags into the rule body:

[0036]

[0037] Among them W u is a trainable weight matrix;

[0038] Multi-hop semantic information is obtained through multiple rounds of information propagation; in the t-th round of information propagation, the vector representation of the rule subject is as follows:

[0039]

[0040] By calculating the similarity between the rule body and the rule label, the label with the highest similarity is selected as the label of the target rule. mod (r i ):

[0041]

[0042] To optimize the trainable parameters and matrices in neighbor information propagation and public information propagation, a loss function is designed and calculated based on the labeled dataset and seed rule set. The parameters and matrices are optimized using the stochastic gradient descent method to minimize the loss, enabling the model to correctly represent the rule subject and rule label vector, thereby improving the quality of rule generation.

[0043] The loss function is set as follows:

[0044] loss total =loss nei +loss com (10)

[0045]

[0046]

[0047] Here, `dis()` is a distance function used to measure the Euclidean distance between two vectors; `loss` nei The aim is to shorten the implied association (P) e The distance between nodes of ) increases (P) c The distance between nodes with opposing relationships; i′m Representation rule r i Authentic labels; m Indicates that except for l′ m Other rule labels besides loss com The aim is to minimize the distance between the rule body and its associated rule tag.

[0048] Preferably, in step 5, the pattern matching is defined as: given rule r i Sentences j When s j .sub-type=r i .sub-type, s j .obj-type=r i .obj-type and Then s j by r i Pattern matching successful; sub-type and obj-type represent the entity types of the subject and object, respectively, and w represents the context word sequence of the text containing the subject and object entities.

[0049] Preferably, in step 6, the semantic matching mechanism based on cross-attention includes sentence-rule SR attention and rule-rule RR attention;

[0050] SR attention is used to adjust the word weights α in a sentence according to rules. k This weakens irrelevant information in the sentence;

[0051]

[0052] Where α k Sentence s j The word w in k The weight, W b b are trainable parameters; For rule r i The vector representation of , which is the average of the rule body and rule label vectors; s j This indicates the sentence to be matched; w k Then it means s j Containing words; For w k Word vectors;

[0053] Sentence s is calculated based on word weight. j semantic vectors

[0054]

[0055] Rule r iand s j The initial matching scores are as follows:

[0056]

[0057] Where Sim() is the cosine similarity function;

[0058] RR attention is used to enrich the semantic information of rules by leveraging a rule knowledge graph. First, it calculates the initial matching score between neighboring rules and the sentence. Then, it weights the initial matching scores based on the semantic association of the rules to obtain the final matching score, Score(r). i ,s j );

[0059] Score(r i ,s j )=hScore′(r i ,s j )+(1-h)Score′(N i ,s j (16)

[0060]

[0061]

[0062] Where h is an adjustment parameter used to adjust the proportion of the initial matching score of the neighbor rule in the final matching score; λ(r i ,p m ,r n ) represents the relation weight, used to aggregate r i The initial matching score of the neighbor rule; W s It is a trainable matrix;

[0063] To improve the accuracy of rule-based semantic matching, the model loss is calculated using seed rules and their matching sentences; the parameters are then optimized using stochastic gradient descent to minimize the loss.

[0064] loss matching =Max(0,[-Score(r)] i ,s j )+Score(r f ,s j )]+π) (19)

[0065] Among them, Score(r i ,s j Score(r) represents the matching score of a rule and the sentence that matches it correctly. f ,s j) represents the matching score of the rule and the sentences that mismatch it; π is a parameter to ensure that there is a difference between the correct matching score and the incorrect matching score;

[0066] The semantic matching is defined as: given rule r i Sentences j When s j .sub-type=r i .sub-type, s j .obj-type=r i .obj-type and matching score Score(r) i ,s j If )≥δ, then s j by r i Semantic matching successful; sub-type and obj-type represent the entity types of the subject and object, respectively.

[0067] Preferably, in step 7, the sentence label is determined by voting among the rules that match the sentence, and the label with the most votes is selected as the sentence label; then, the sentences that have matched a certain rule vote to verify the rule label, and the label with the most votes is selected as the rule label. sen (r i ).

[0068] Preferably, in step 8, the inconsistency-oriented active learning strategy aims to select inconsistent rules based on limited human resources and hand them over to humans for verification of their labels; the verification results will be used to optimize the training of the rule generation model; the inconsistency of the rules can be divided into two main cases: (1) rule generation; the semantic information of the rule label and the rule body is inconsistent; (2) rule matching; the semantic information of the rule is inconsistent with that of the sentence.

[0069] To evaluate rule inconsistencies, we first use contrastive learning to calculate the semantic distance between rules and between rules and sentences; then, we use data augmentation to obtain vector representations of different rules and sentences; finally, we design a contrastive learning function that makes vectors of the same rule / sentence closer together and vectors of different rules / sentences farther apart.

[0070]

[0071] in Let τ be the indicator function, 2N be the number of rules and sentences after data augmentation, and e be the expression. i and e j Vectors of the same rules / sentences; e i and e k Vectors representing different rules / sentences;

[0072] Next, based on the representations of rules and sentences, an active learning function is designed to evaluate rule inconsistencies; by comparing the semantic distance between the rule and its neighboring rules, inconsistencies in rule generation are evaluated.

[0073]

[0074] Where N con (r i ) and N ent (r i Neighbor rules with opposing and implied labels; Min() aims to find the minimum semantic distance between a rule and its implied rules, and Max() aims to find the maximum semantic distance between a rule and its opposing rules;

[0075] Inconsistencies in rule matching are evaluated by comparing the semantic distance between the rules and the matched sentences.

[0076]

[0077] Where S c (r i ) and S w (r i The numbers represent the sentences that are correctly matched and those that are incorrectly matched; Min() aims to find the minimum semantic distance between the rule and the correctly matched sentences, while Max() aims to find the maximum semantic distance between the rule and the incorrectly matched sentences.

[0078] By integrating the inconsistency scores from the rule generation and rule matching stages, rules requiring manual verification are identified.

[0079]

[0080] The argmax() function aims to find the rule with the highest inconsistency score.

[0081] The technical solution adopted by the system of this invention is: a weakly supervised data annotation rule discovery system based on knowledge graphs, characterized in that it includes:

[0082] One or more processors;

[0083] A storage device for storing one or more programs, which, when executed by one or more processors, enable the one or more processors to implement the knowledge graph-based weakly supervised data annotation rule discovery method.

[0084] The innovative aspects of this invention include:

[0085] (1) In order to reduce the number of misclassified rules, this invention designs a label-aware rule generation method, which propagates prior knowledge from seed rules to candidate rules based on rule knowledge graph.

[0086] (2) To reduce the number of incorrectly matched sentences, this invention proposes a semantic matching mechanism based on cross-attention, which refines the semantic information of sentences based on rule knowledge graphs while enriching the semantic information of rules.

[0087] (3) In order to improve the overall quality of rule discovery, this invention proposes an inconsistency-oriented active learning strategy to verify inconsistent rules in rule generation and matching.

[0088] Compared with existing methods, the present invention has the following beneficial effects:

[0089] (1) This invention optimizes rule generation and rule matching based on multi-dimensional semantic association between rules, which alleviates the semantic drift problem in rule discovery and provides interpretability for rule discovery.

[0090] (2) This invention reveals the potential of using rule knowledge graphs to construct high-quality rule sets. Since this method uses limited prior knowledge, it can be applied to highly specialized fields.

[0091] (3) The rules discovered by this invention can not only extract knowledge triples, but also mine knowledge associations, discover text patterns and reveal text semantic features. Attached Figure Description

[0092] The technical solutions described herein are further illustrated below using examples and specific implementation methods. Additionally, accompanying drawings are used in the description of the technical solutions. Those skilled in the art can, without any creative effort, obtain other drawings and the intent of the present invention based on these drawings.

[0093] Figure 1 : This is a flowchart of a method according to an embodiment of the present invention.

[0094] Figure 2 Example of a rule knowledge graph provided in an embodiment of the present invention.

[0095] Figure 3 This is an example of rule generation for label awareness provided in an embodiment of the present invention.

[0096] Figure 4 This is an example of rule-based semantic matching based on cross-attention provided in an embodiment of the present invention. Detailed Implementation

[0097] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0098] This invention aims to construct a rule knowledge graph by utilizing the semantic relationships between rules, and to discover high-quality data annotation rules based on the rule knowledge graph, thereby alleviating the semantic drift problem in rule discovery. This invention faces the following challenges: (1) In rule generation, multidimensional semantic relationships between rules may provide conflicting semantic constraints for rule labels. (2) In rule matching, irrelevant semantic information contained in sentences can affect the similarity calculation between rules and sentences. (3) Due to limited human resources, it is difficult to verify rules, which may affect the quality of newly discovered rules.

[0099] Please see Figure 1 This embodiment provides a method for discovering weakly supervised data annotation rules based on knowledge graphs, including the following steps:

[0100] Step 1: Select a small amount of data from the corpus for annotation, and divide the corpus into an annotated dataset S. seed and unlabeled dataset C;

[0101] In one implementation, the data labeling rules are defined by the rule subject r. i The mapping formed by .b ​​and rule label l: r i .b→l; where the rule subject r i .b is a text sequence, represented as p = [w, sub-type, w, obj-type, w], where sub-type and obj-type represent the entity types of the subject and object, respectively, and w represents the context word sequence of the text containing the subject and object entities.

[0102] Step 2: By mining the labeled dataset S seed The longest common subsequence of the unlabeled dataset C is used to construct a seed rule set r; the longest common subsequence of the unlabeled dataset C is used to construct a candidate rule set R. can ;

[0103] In one implementation, mining the longest common subsequence refers to, after preprocessing the corpus by removing stop words and punctuation marks, comparing multiple corpus sequences to find the longest common subsequence containing both a subject and an object, which serves as the basis for the rules. The mining of the longest common subsequence continues based on the results until no new sequences can be found.

[0104] Step 3: In the seed rule set R and candidate rule set R can In the process, we mine the multidimensional semantic relationships between rules and construct a rule knowledge graph G;

[0105] In one implementation, the rule knowledge graph G = {(n i ,p m ,nj )|n i ,n j ∈V,p m ∈P}; where V=V L ∪V R.b V is a set of nodes. L and V R.b Let P represent the set of rule body nodes and the set of rule label nodes, respectively, where P = P B-L ∪P B-B ∪P L-L Let P be the set of edges. B-L ={(n i ,p m ,n j )|n i ∈V R.b ,m j ∈V L} represents the semantic relationship between the rule body and the rule tag, P B-B ={(n i ,p m ,n j )|n i ,n j ∈V R.b} represents the semantic relationship between rule subjects, P L-L ={(n i ,p m ,n j )|n i ,n j ∈V L} represents the semantic relationship between rule tags, n i ,n j Let p represent two nodes in the node set V. m For two nodes n i ,n j The edge between;

[0106] The multidimensional semantic associations between the rules refer to "belonging to," "implication," and "opposition" associations; the semantic association between the rule subject and the rule label is a "belonging to" association, indicating that the rule subject and the rule label contain the same semantic information; the associations between rule subjects and between rule labels include "implication" and "opposition" associations; an "implication" association indicates that there is the same semantic information between the premise and the hypothesis, and the hypothesis is derived from the premise, that is... When v i Contains v j Then v i True v j True; the "opposite" relation indicates that there is contradictory semantic information between the premise and the hypothesis, and the hypothesis cannot be derived from the premise, that is... When v i Contains v j Then v i True v j False; adjust and expand the semantic association of the rules according to the data annotation requirements.

[0107] Step 4: In the rule knowledge graph G, use the label-aware rule generation mechanism to infer the candidate rule set R. can The rule in the tag L mod (r i );

[0108] In one implementation, the label-aware rule generation mechanism includes initial embedding, neighbor information propagation, and public information propagation;

[0109] The initial embedding is achieved by using the pre-trained language model BERT to generate vector representations of rule labels and rule bodies. and in The vector representing the label of the m-th rule; The vector representing the rule body of the i-th rule;

[0110] The neighbor information propagation is used to propagate semantic information between rule subjects or rule tags; among rule subjects, given a specific target rule subject r... i .b,N i ={(r i .b,p m ,r n .b)|(r i .b,p m ,r n b)∈G} contains r i All neighboring rule subjects of .b; then the semantic information of these neighboring rule subjects can be aggregated in the following way:

[0111]

[0112]

[0113]

[0114] Where, θ(r) i .b,p m ,r n b) represents the trainable weight parameters, indicating the neighbor rule subject r. n .b and the target rule subject r i The relationship between .b and p m The weights they represent; For relation pm ; It is a trainable weight matrix, where d and d′ are the dimensions of the matrix; Then it represents the set of neighbor rule subjects N. i The vector; when θ(r) i .b,p m ,r n The larger the value of .b), the stronger the individual neighbor rule subject vector. In calculating the vector of the neighbor rule subject set The greater the contribution, the better;

[0115] The semantic information contained in the neighbor rule body is incorporated into the target rule body to obtain the vector representation of the target rule body:

[0116]

[0117] Where f() is the activation function, W r The weight matrix is ​​trainable; similarly, the vector representation of the target rule label is updated by propagating neighbor information in the rule labels.

[0118] The aforementioned public information dissemination occurs between the rule subject and the rule tag, integrating the semantic information of the rule subject and the rule tag, and incorporating the rule subject information into the rule tag:

[0119]

[0120]

[0121] Where σ(l) m ,r i b) is a trainable weight parameter used to represent how much public information is passed from the rule subject to the rule label; B, W l and W c For trainable model parameters; l m This represents the m-th rule label; Indicates that it belongs to the rule label l m The set of rules;

[0122] Next, to enrich the vector representation of the rule body, the semantic information of the rule label is integrated into the rule body:

[0123]

[0124] Among them W u is a trainable weight matrix;

[0125] Through multiple rounds of information propagation, multi-hop semantic information can be obtained; in the t-th round of information propagation, the vector representation of the rule subject can be defined as:

[0126]

[0127] By calculating the similarity between the rule body and the rule label, the label with the highest similarity is selected as the label of the target rule. mod (r i ):

[0128]

[0129] To optimize the trainable parameters and matrices in neighbor information propagation and public information propagation, a loss function is designed and calculated based on the labeled dataset and seed rule set. The parameters and matrices are optimized using the stochastic gradient descent method to minimize the loss, enabling the model to correctly represent the rule subject and rule label vector, thereby improving the quality of rule generation.

[0130] The loss function is set as follows:

[0131] loss total =loss nei +loss com (10)

[0132]

[0133]

[0134] Here, `dis()` is a distance function used to measure the Euclidean distance between two vectors. (Loss) nei The aim is to shorten the implied association (P) e The distance between nodes of ) increases (P) c The distance between nodes that have an opposing relationship. m Representation rule r i The true label. m Indicates that except for l′ m Other rule labels besides loss. com The aim is to minimize the distance between a rule and its actual label.

[0135] Step 5: In the rule knowledge graph G, the rules are converted into regular expressions to achieve pattern matching of the rules against the corpus;

[0136] In one implementation, the pattern matching is defined as: given rule r i Sentences j When s j .sub-type=ri .sub-type, s j .obj-type=r i .obj-type and Then s j by r i Pattern matching successful; sub-type and obj-type represent the entity types of the subject and object, respectively, and w represents the context word sequence of the text containing the subject and object entities.

[0137] Step 6: In the rule knowledge graph G described in step 3, the semantic matching of rules to the corpus is achieved through a semantic matching mechanism based on cross-attention;

[0138] In one implementation, the cross-attention-based semantic matching mechanism includes sentence-rule SR attention and rule-rule RR attention;

[0139] SR attention is used to adjust the word weights α in a sentence according to rules. k This weakens irrelevant information in the sentence; the formula for calculating word weight is as follows:

[0140]

[0141] Where α k Sentence s j The word w in k The weight, W b b are trainable parameters; For rule r i The vector representation of , which is the average of the rule body and rule label vectors; s j This indicates the sentence to be matched; w k Then it means s j Containing words; For w k Word vectors;

[0142] Then, sentence s can be calculated based on word weights. j semantic vectors

[0143]

[0144] Rule r i and s j The initial matching scores are as follows:

[0145]

[0146] Where Sim() is the cosine similarity function;

[0147] RR attention is used to enrich the semantic information of rules by leveraging a rule knowledge graph. First, this embodiment calculates the initial matching score between neighboring rules and the sentence. Then, it weights the initial matching score based on the semantic association of the rules to obtain the final matching score, Score(r). i ,s j );

[0148] Score(r i ,s j )=hScore′(r i ,s j )+(1-h)Score′(N i ,s j (16)

[0149]

[0150]

[0151] Where h is an adjustment parameter used to adjust the proportion of the initial matching score of the neighbor rule in the final matching score; λ(r i ,p m ,r n ) represents the relation weight, used to aggregate r i The initial matching score of the neighbor rule; W s It is a trainable matrix;

[0152] To improve the accuracy of rule-based semantic matching, the model loss is calculated using seed rules and their matching sentences; the parameters are then optimized using stochastic gradient descent to minimize the loss.

[0153] loss matching =Max(0,[-Score(r)] i ,s j )+Score(r f ,s j )]+π) (19)

[0154] Among them, Score(r i ,s j Score(r) represents the matching score of a rule and the sentence that matches it correctly. f ,s j ) represents the matching score of the rule and the sentences that mismatch it; π is a parameter to ensure that there is a difference between the correct matching score and the incorrect matching score;

[0155] The semantic matching is defined as: given rule r i Sentences j When s j .sub-type=ri .sub-type, s j .obj-type=r i .obj-type and matching score Score(r) i ,S j If )≥δ, then S j R i Semantic matching successful; sub-type and obj-type represent the entity types of the subject and object, respectively.

[0156] Step 7: In the pattern matching and semantic matching results described in steps 5 and 6, select the tag with the most matches as the sentence tag to obtain the matching sentence set S. t Select the sentence tag with the most matching occurrences as the rule tag L. sen (r i );

[0157] In one implementation, the sentence label is determined by voting among the rules that match the sentence, and the label with the most votes is selected as the sentence label; then, the sentences that have matched a certain rule vote to verify the rule label, and the label with the most votes is selected as the rule label L. sen (R i ).

[0158] Step 8: In the rule knowledge graph described in Step 4 and the matching sentence set described in Step 7, rules whose rule generation and rule matching performance are consistent are regarded as confidence rules, i.e., L mod (R i )=l sen (R i Using an inconsistency-oriented active learning strategy, rules that are inconsistent in rule generation and rule matching are selected for manual verification, and the confidence rules and manual verification results are added to the rule set r.

[0159] In one implementation, the inconsistency-oriented active learning strategy aims to select inconsistent rules based on limited human resources and have them verified by humans; the verification results will be used to optimize the training of the rule generation model; the inconsistency of rules can be divided into two main cases: (1) Rule generation; the semantic information of the rule label and the rule subject is inconsistent; for example, the semantic similarity of rule subjects belonging to the same or similar rule labels is low. The semantic similarity of rule subjects belonging to opposing rule labels is high. (2) Rule matching; the semantic information of the rule and the sentence is inconsistent; for example, the rule matches sentences belonging to opposing labels.

[0160] To evaluate rule inconsistencies, we first use contrastive learning to calculate the semantic distance between rules and between rules and sentences; then we use data augmentation to enhance the vector representations of rules and sentences; finally, we design a contrastive learning function that makes vectors of the same rule / sentence closer together and vectors of different rules / sentences farther apart.

[0161]

[0162] in Let τ be the indicator function, 2N be the number of rules and sentences after data augmentation, and e be the expression. i and e j Vectors of the same rules / sentences; e i and e k Vectors representing different rules / sentences;

[0163] Next, based on the representations of rules and sentences, an active learning function is designed to evaluate rule inconsistencies; by comparing the semantic distance between the rule and its neighboring rules, inconsistencies in rule generation are evaluated.

[0164]

[0165] Where N con (r i ) and N ent (r i Neighbor rules with opposing and implied labels; Min() aims to find the minimum semantic distance between a rule and its implied rules, and Max() aims to find the maximum semantic distance between a rule and its opposing rules;

[0166] Inconsistencies in rule matching can be assessed by comparing the semantic distance between the rule and the matched sentence:

[0167]

[0168] Where S c (r i ) and S w (r i The numbers represent the sentences that are correctly matched and those that are incorrectly matched; Min() aims to find the minimum semantic distance between the rule and the correctly matched sentences, while Max() aims to find the maximum semantic distance between the rule and the incorrectly matched sentences.

[0169] The following formula can be used to integrate the inconsistency scores from the rule generation and rule matching stages to identify rules that require manual verification:

[0170]

[0171] The argmax() function aims to find the rule with the highest inconsistency score.

[0172] Step 9: Repeat steps 3-8. When the manual annotation cost is exhausted, stop the iteration and output the rule set R and the matching sentence set S. t .

[0173] This embodiment can be implemented using software, and the overall algorithm is as follows:

[0174]

[0175]

[0176] The invention will be further illustrated below through specific experiments.

[0177] Please see Figure 2 , Figure 3 and Figure 4, which are respectively the example of the rule knowledge graph provided by this experiment, the example of label-aware rule generation provided by this experiment, and the example of rule semantic matching based on cross-attention provided by this experiment. In this experiment, seed rules r1 (person, person, stranger -> stranger), r3 (person, gave birth to, person -> mother) and candidate rule r2 (person, mother, person ->?) are mined from the corpus. The rule label of the candidate rule is unknown. The predefined rule labels include: l1 stranger, l2 mother, l3 relative. Based on the mined seed rules and candidate rules, a rule knowledge graph is constructed using the multi-dimensional semantic associations between the rules. The triples included in this rule knowledge graph are as follows: <r1.b, opposite, r3.b>, <r1.b, opposite, r2.b>, <r2.b, implies, r3.b>, <l1, opposite, l2>, <l1, opposite, l3>, <l2, opposite, l3>, <r1.b, belongs to, l1>, <r3.b, belongs to, l2>. Based on the rule knowledge graph, the label of the candidate rule r2 is inferred by the label-aware rule generation method. Specifically, first, neighbor information propagation is implemented between rule bodies and between rule labels to transfer the semantic information of r1.b and r3.b to r2.b. At the same time, the semantic information of l1 and l3 is transferred to l2. Then, information propagation between the rule body and the label is achieved through common information propagation, transferring the semantic information of r3.b to l2 and the semantic information of r1.b to l1. Through multiple rounds of neighbor information propagation and common information propagation, the embedding vectors of r2.b, l1 and l2 are obtained. Since the similarity between r2.b and l2 is higher than the similarity between r2.b and l1, it can be determined that the label of r2 is l2. Based on the predicted rules, pattern matching and semantic matching based on cross-attention are performed. When calculating whether s1 and r3 can be semantically matched, first calculate the similarity between each word in s1 and r3. Since the similarity between "gave birth to" and r3 is relatively high, the weight of this word is relatively high. Using the word weights, the sentence vector of s1 can be calculated based on the word vectors. By calculating the vector similarity between s1 and r3, the initial semantic matching score between s1 and r3 can be obtained. Then, calculate the initial semantic matching scores of the neighbor rules r1, r2 of r3 and s1 respectively, and aggregate the matching scores using the semantic associations between these three rules. Since r2 and r3 are in an implication relationship, the matching score of r2 accounts for a relatively high proportion. Finally, the matching score Score(r3, s1) = 0.587 between s1 and r3 is obtained. Since Score(r3, s1) is higher than the specified threshold of 0.5, s1 can be semantically matched by r3. Since s1 is simultaneously matched by r1, r2, r3, and both r2 and r3 represent the "mother" relationship, the label of s1 is "mother". From s1, the label of r1 can be inferred as "mother". Then the labels of r1 in the rule generation stage and the rule matching stage are inconsistent. Therefore, this experiment will use the inconsistency-oriented active learning strategy to select the inconsistent rules for human verification.For r1, we first compare the semantic distances of its neighboring rules r2 and r3 with r1 to evaluate the inconsistency of r1 during the rule generation stage. Specifically, since r2 and r3 are both implication relations with r1, and dis(r1,r2)>dis(r1,r3), RC(r1)=dis(r1,r3)=0.124. Similarly, in rule matching, s1 is a sentence that r1 mismatched, therefore... Therefore, the final inconsistency score of r1 is 0.939. All rules are sorted from highest to lowest inconsistency score, and the top 5% of rules are selected for human verification. The results of human verification will be used to update the rule knowledge graph.

[0178] This embodiment also provides a weakly supervised data annotation rule discovery system based on knowledge graphs, including:

[0179] One or more processors;

[0180] A storage device for storing one or more programs, which, when executed by one or more processors, enable the one or more processors to implement the knowledge graph-based weakly supervised data annotation rule discovery method.

[0181] The present invention proposes a weakly supervised data annotation rule discovery method based on knowledge graphs, which solves the semantic drift problem in data annotation under weak supervision. Applied to the field of natural language processing, it can discover rules from unstructured text based on a small set of rule-annotated samples, improve the accuracy of rule generation by utilizing multi-dimensional semantic association of rules, improve the rule matching recall by utilizing semantic matching mechanism, and use active learning strategy to verify rules, ultimately generating high-quality data annotation rules.

[0182] It should be understood that the above description of the preferred embodiments is quite detailed, but it should not be considered as a limitation on the scope of protection of this invention. Those skilled in the art, under the guidance of this invention, can make substitutions or modifications without departing from the scope of protection of the claims of this invention, and all such substitutions or modifications fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.

Claims

1. A method for discovering weakly supervised data annotation rules based on knowledge graphs, characterized in that, Includes the following steps: Step 1: Select a certain amount of data from the corpus for annotation, and divide the corpus into an annotated dataset S. seed and unlabeled dataset C; Step 2: By mining the labeled dataset S seed The longest common subsequence of the unlabeled dataset C is used to construct a seed rule set R; the longest common subsequence of the unlabeled dataset C is used to construct a candidate rule set R. can ; Step 3: In the seed rule set R and candidate rule set R can In the process, we mine the multidimensional semantic relationships between rules and construct a rule knowledge graph G; Step 4: In the rule knowledge graph G, infer the candidate rule set R based on the semantic relationships between rules. can The rule in the tag L mod (r i ); Step 5: In the rule knowledge graph G, the rules are converted into regular expressions to achieve pattern matching of the rules against the corpus; Step 6: In the rule knowledge graph G described in step 3, the semantic matching of rules to the corpus is achieved through a semantic matching mechanism based on cross-attention; Step 7: In the pattern matching and semantic matching results described in steps 5 and 6, select the tag with the most matches as the sentence tag to obtain the matching sentence set S. t Select the sentence tag with the most matching occurrences as the rule tag L. sen (r i ); Step 8: In the rule knowledge graph described in Step 4 and the matching sentence set described in Step 7, rules whose rule generation and rule matching performance are consistent are regarded as confidence rules, i.e., L mod (r i ) = L sen (r i Using an inconsistency-oriented active learning strategy, rules that are inconsistent in rule generation and rule matching are selected for manual verification, and the confidence rules and manual verification results are added to the rule set R. Step 9: Repeat steps 3-8. When the manual annotation cost is exhausted, stop the iteration and output the rule set R and the matching sentence set S. t .

2. The method for discovering weakly supervised data annotation rules based on knowledge graphs according to claim 1, characterized in that: In step 1, the data labeling rules are defined by the rule subject r. i The mapping formed by .b ​​and rule label l: r i .b→l; where the rule subject r i .b is a text sequence, represented as p = [w, sub-type, w, obj-type, w], where sub-type and obj-type represent the entity types of the subject and object, respectively, and w represents the context word sequence of the text containing the subject and object entities.

3. The method for discovering weakly supervised data annotation rules based on knowledge graphs according to claim 1, characterized in that: In step 2, the mining of the longest common subsequence involves removing stop words and punctuation marks from the corpus, comparing multiple corpus sequences, and mining the longest common subsequence containing the subject and object as the main rule; the mining of the longest common subsequence continues based on the mining results until no new sequence can be mined.

4. The method for discovering weakly supervised data annotation rules based on knowledge graphs according to claim 1, characterized in that: In step 3, the rule knowledge graph G = {(n i p m n j )|n i n j ∈V, p m ∈P}; where V=V L ∪V R.b V is a set of nodes. L and V R.b Let P represent the set of rule body nodes and the set of rule label nodes, respectively, where P = P B-L ∪P B-B ∪P L-L Let P be the set of edges. B-L ={(n i p m n j )|n i ∈V R.b n j ∈V L ) represents the semantic relationship between the rule body and the rule tag, P B-B ={(n i p m n j )|n i n j ∈V R.b } represents the semantic relationship between rule subjects, P L-L ={(n i p m n j )|n i n j ∈V L } represents the semantic relationship between rule tags, n i n j Let p represent two nodes in the node set V. m For two nodes n i n j The edge between; The multidimensional semantic associations between the rules refer to "belonging to," "implication," and "opposition" associations; the semantic association between the rule subject and the rule label is a "belonging to" association, indicating that the rule subject and the rule label contain the same semantic information; the associations between rule subjects and between rule labels include "implication" and "opposition" associations; an "implication" association indicates that there is the same semantic information between the premise and the hypothesis, and the hypothesis is derived from the premise, that is... When v i Contains v j Then v i True v j True; the "opposite" relation indicates that there is contradictory semantic information between the premise and the hypothesis, and the hypothesis cannot be derived from the premise, that is... When v i Contains v j Then v i True v j False; adjust and expand the semantic association of the rules according to the data annotation requirements.

5. The method for discovering weakly supervised data annotation rules based on knowledge graphs according to claim 1, characterized in that: In step 4, a label-aware rule generation mechanism is used to infer the candidate rule set R based on the relationships between rules. can The rule in the tag L mod (r i ); The label-aware rule generation mechanism includes initial embedding, neighbor information propagation, and public information propagation; The initial embedding is achieved by using the pre-trained language model BERT to generate vector representations of rule labels and rule bodies. and in The vector representing the label of the m-th rule; The vector representing the rule subject of the i-th rule; The neighbor information propagation is used to propagate semantic information between rule subjects or rule tags; among rule subjects, given a specific target rule subject r... i .b, N i ={(r i .b, p m r n .b)l(r i .b, p m r n b)∈G} contains r i All neighboring rule subjects of .b; then the semantic information of these neighboring rule subjects can be aggregated in the following way: Where, θ(r) i .b, p m r n b) represents the trainable weight parameters, indicating the neighbor rule subject r. n .b and target rule subject r i The relationship between .b and p m The weights they represent; For relation p m ; It is a trainable weight matrix, where d and d′ are the dimensions of the matrix; Then it represents the set of neighbor rule subjects N. i The vector; when θ(r) i .b, p m r n The larger the value of .b), the stronger the individual neighbor rule subject vector. In calculating the vector of the neighbor rule subject set The greater the contribution, the better; Integrate the semantic information contained in the neighboring rule body into the target rule body, and update the vector representation of the target rule body: Where f() is the activation function, W r The weight matrix is ​​trainable; similarly, the vector representation of the target rule label is updated by propagating neighbor information in the rule labels. The aforementioned public information dissemination occurs between the rule subject and the rule tag, integrating the semantic information of the rule subject and the rule tag, and incorporating the rule subject information into the rule tag: Where σ(l) m r i b) is a trainable weight parameter used to represent how much public information is passed from the rule subject to the rule label; B, W l and W c For trainable model parameters; l m This represents the m-th rule label; Indicates that it belongs to the rule label l m The set of rules; Integrate the semantic information of the rule tags into the rule body: Among them W u is a trainable weight matrix; Multi-hop semantic information is obtained through multiple rounds of information propagation; in the t-th round of information propagation, the vector representation of the rule subject is as follows: By calculating the similarity between the rule body and the rule label, the label with the highest similarity is selected as the label of the target rule. mod (r i ): To optimize the trainable parameters and matrices in neighbor information propagation and public information propagation, a loss function is designed and calculated based on the labeled dataset and seed rule set. The parameters and matrices are optimized using the stochastic gradient descent method to minimize the loss, enabling the model to correctly represent the rule subject and rule label vector, thereby improving the quality of rule generation. The loss function is set as follows: loss total =loss nei +loss com (10) Here, `dis()` is a distance function used to measure the Euclidean distance between two vectors; `loss` nei The aim is to shorten the implied association (P) e The distance between nodes of ) increases (P) c The distance between nodes with opposing relationships; l′ m Representation rule r i Authentic labels; m Indicates that except for l′ m Other rule labels besides loss com The aim is to minimize the distance between the rule body and its associated rule tag.

6. The method for discovering weakly supervised data annotation rules based on knowledge graphs according to claim 1, characterized in that: In step 5, the pattern matching is defined as: given rule r i Sentences j When s j .sub-type=r i .sub-type, s j .obj-type=r i .obj-type and Then s j by r i Pattern matching successful; sub-type and obj-type represent the entity types of the subject and object, respectively, and w represents the context word sequence of the text containing the subject and object entities.

7. The method for discovering weakly supervised data annotation rules based on knowledge graphs according to claim 1, characterized in that: In step 6, the semantic matching mechanism based on cross-attention includes sentence-rule SR attention and rule-rule RR attention; SR attention is used to adjust the word weights α in a sentence according to rules. k This weakens irrelevant information in the sentence; Where α k Sentence s j The word w in k The weights, W b b are trainable parameters; For rule r i The vector representation of , which is the average of the rule body and rule label vectors; s j This indicates the sentence to be matched; w k Then it means s j Contains words; For w k Word vectors; Sentence s is calculated based on word weight. j semantic vectors Rule r i and s j The initial matching scores are as follows: Where Sim() is the cosine similarity function; RR attention is used to enrich the semantic information of rules by leveraging a rule knowledge graph. First, it calculates the initial matching score between neighboring rules and the sentence. Then, it weights the initial matching scores based on the semantic association of the rules to obtain the final matching score, Score(r). i s j ); Score(r i ,s j )=hScore′(r i ,s j )+(1-h)Score′(N i ,s j ) (16) Where h is an adjustment parameter used to adjust the proportion of the initial matching score of the neighbor rule in the final matching score; λ(r i p m r n ) represents the relation weight, used to aggregate r i The initial matching score of the neighbor rule; W s It is a trainable matrix; To improve the accuracy of rule-based semantic matching, the model loss is calculated using seed rules and their matching sentences; the parameters are then optimized using stochastic gradient descent to minimize the loss. loss matching =Max(0,[-Score(r i ,s j )+Score(r f ,s j )]+π) (19) Among them, Score(r i s j Score(r) represents the matching score of a rule and the sentence that matches it correctly. f s j ) represents the matching score of the rule and the sentences that mismatch it; π is a parameter to ensure that there is a difference between the correct matching score and the incorrect matching score; The semantic matching is defined as: given rule r i Sentences j When s j· sub-type = r i .sub-type, s j .obj-type=r i .obj-type and matching score Score(r) i s j If )≥δ, then s j by r i Semantic matching successful; sub-type and obj-type represent the entity types of the subject and object, respectively.

8. The method for discovering weakly supervised data annotation rules based on knowledge graphs according to claim 1, characterized in that: In step 7, the sentence label is determined by voting among the rules that match the sentence; the label with the most votes is selected as the sentence label. Next, sentences that have matched a rule vote to verify the rule label, and the label with the most votes is selected as the rule label. sen (r i ).

9. The method for discovering weakly supervised data annotation rules based on knowledge graphs according to claim 1, characterized in that: In step 8, the inconsistency-oriented active learning strategy aims to select inconsistent rules based on limited human resources and hand them over to humans to verify their labels. The verification results will be used to optimize the training of the rule generation model; rule inconsistency can be categorized into two main scenarios: (1) rule generation: the semantic information of the rule label and the rule body is inconsistent; (2) rule matching: the semantic information of the rule is inconsistent with that of the sentence. To evaluate rule inconsistencies, we first use contrastive learning to calculate the semantic distance between rules and between rules and sentences; then, we use data augmentation to obtain vector representations of different rules and sentences; finally, we design a contrastive learning function that makes vectors of the same rule / sentence closer together and vectors of different rules / sentences farther apart. in Let τ be the indicator function, 2N be the number of rules and sentences after data augmentation, and e be the expression. i and e j Vectors of the same rules / sentences; e i and e k Vectors representing different rules / sentences; Next, based on the representations of rules and sentences, an active learning function is designed to evaluate rule inconsistencies; by comparing the semantic distance between the rule and its neighboring rules, inconsistencies in rule generation are evaluated. Where N con (r i ) and N ent (r i Neighbor rules with opposing and implied labels; Min() aims to find the minimum semantic distance between a rule and its implied rules, and Max() aims to find the maximum semantic distance between a rule and its opposing rules; Inconsistencies in rule matching are evaluated by comparing the semantic distance between the rules and the matched sentences. Where S c (r i ) and S w (r i The numbers represent the sentences that are correctly matched and those that are incorrectly matched; Min() aims to find the minimum semantic distance between the rule and the correctly matched sentences, while Max() aims to find the maximum semantic distance between the rule and the incorrectly matched sentences. By integrating the inconsistency scores from the rule generation and rule matching stages, rules requiring manual verification are identified. The argmax() function aims to find the rule with the highest inconsistency score.

10. A weakly supervised data annotation rule discovery system based on knowledge graphs, characterized in that, Its features include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the knowledge graph-based weakly supervised data annotation rule discovery method as described in any one of claims 1 to 9.

Images