A data annotation rule discovery method and device for entity relationship extraction

Abstract

The present invention discloses a data annotation rule discovery method and device for entity relationship extraction, wherein the method defines data annotation rules (PN rules) to represent positive rules and negative rules and their importance scores. PN rules can be used to jointly filter out incorrectly matched sentences to improve the accuracy of rule matching; a rule semantic matching mechanism based on a word-sentence association graph is designed, and a random walk algorithm guided by semantic similarity probability is used according to the semantic association between rules, words and sentences to improve the rule matching recall rate. Matching conflicts of PN rules will be resolved through a weighted voting strategy; a collaborative training rule verification mechanism is proposed to further verify the sentences matched by the rules, thereby improving the quality of iterative rule discovery. The present invention can solve the problem of overfitting of entity relationship extraction models caused by insufficient training data, and iteratively generate data annotation rules under limited prior knowledge to improve data annotation quality.

Description

Technical Field

[0001] The present invention relates to the technical field of entity relationship extraction, and in particular to a method and device for discovering data annotation rules for entity relationship extraction. Background Art

[0002] The task of entity relationship extraction involves semantically identifying the relationships between pairs of entities in text. Deep learning models are prone to overfitting when there is little labeled data. Therefore, data annotation plays a crucial role in entity relationship extraction.

[0003] Data annotation for entity relationship extraction involves assigning relational labels to sentences to be annotated. Existing data annotation methods generally fall into two categories: one is to manually define rules and annotate the corpus, such as Snorkel. Due to the limited domain knowledge held by experts, manually defined rules are prone to duplication and incomplete coverage. The other approach involves expert annotation of the corpus and automatic rule mining. However, due to the ambiguity and polysemy of natural language, the problem of low-quality rules caused by mismatches has not been adequately addressed. For example, TruePIE, proposed by Q. Li et al., uses positive and negative rules to match positive and negative examples respectively, and expands the rule set through an iterative mechanism. However, this method lacks a mechanism for resolving conflicts between positive and negative rules, resulting in reduced rule matching accuracy. NERO, proposed by W. Zhou et al., uses word embeddings to calculate the similarity between rules and sentences to achieve soft rule matching, improving the recall rate of rule matching. However, it ignores the semantic connections between rules, words, and sentences, which leads to the introduction of additional mismatched sentences.

[0004] It can be seen from this that the methods in the existing technology have technical problems such as low data labeling accuracy and recall rate. Summary of the Invention

[0005] The present invention provides a data labeling rule discovery method and device for entity relationship extraction, which is used to solve or at least partially solve the technical problem of poor data labeling effect (i.e., low data labeling accuracy and recall rate) in the prior art.

[0006] In order to solve the above technical problems, the first aspect of the present invention provides a data annotation rule discovery method for entity relationship extraction, comprising:

[0007] S1: Divide the preset seed dataset into training set and test set;

[0008] S2: Define PN rules. Based on the defined PN rules, use the frequent pattern mining algorithm to mine the divided training set and build a rule set. The PN rules are composed of body, tag and I t (ri ) constitutes a mapping: r i .body→(r i .tag,I t (r i ), the rule body is a text pattern sequence of p = [w,@sub-type,w,@obj-type,w], where sub-type and obj-type represent the entity types of the subject and object respectively, w represents the context word sequence in the sentence where the subject entity and object entity are located, the tag represents the positive or negative of the rule, and the importance score I t (r i ) is used to determine the positive or negative nature of the rule;

[0009] S3: Convert the rules contained in the rule set into regular expressions, use the converted rules to perform pattern matching on sentences in the preset corpus, and obtain sentences that successfully match the rule pattern;

[0010] S4: Based on the rule set, construct a word-sentence semantic association graph G(V,E) and a transition probability matrix M, and semantically match the rules to the sentences in the preset corpus based on the random walk algorithm guided by semantic similarity probability to obtain sentences that successfully match the semantics of the rules. The word-sentence semantic association graph G(V,E) is an undirected simple graph, V represents the set of nodes, E represents the set of edges, E is a subset of V×V, and the node set V=W * ∪ * , W * represents the word set generated from the rules and candidate sentences, S * The candidate sentence set consists of sentences with the same subject and object entity types. The elements in the transition probability matrix are the probabilities of randomly walking from one node to another:

[0011] S5: Based on the sentences that successfully match the rule pattern and the sentences that successfully match the rule semantics, the conflict resolution mechanism is used to resolve conflicts and construct the target matching sentence set;

[0012] S6: Based on the training set, two machine learning models (PC and SC) are collaboratively trained using a collaborative training method to obtain two trained classification models. The two trained classification models are then used to predict sentences in the target matching sentence set to obtain prediction results, which are added to the training set. The prediction results are the pattern features learned by the machine learning model PC and the semantic features learned by the machine learning model SC.

[0013] S7: Repeat steps S2 to S6. When the frequent pattern mining algorithm cannot mine new rules, stop iteration and output the rule set and the target matching sentence set. The sentences contained in the target matching sentence set and their corresponding labels are the data annotation results. The data annotation results can be used as the training set of the entity relationship extraction model.

[0014] In one embodiment, the importance score of the PN rule in step S2 is calculated as follows:

[0015]

[0016] Where σ is the normalization parameter, is the reliability of the rule, is the regular frequency, is the redundancy of rules;

[0017] The reliability of a rule is calculated as:

[0018]

[0019] Representation rule r i In the tth iteration, the sentences whose true labels are matched are positive. Representation rule r i The sentences whose true labels matched in the tth iteration are negative; and Representation rule r i The sentence set obtained by pattern matching and semantic matching, when When , the rule is a positive rule; when When , the rule is a negative rule. This means that the rule is invalid in round t;

[0020] The rule frequency is calculated as:

[0021]

[0022] and Indicates that r in the tth iteration i Pattern matching and semantic matching and the actual label is the same as r i .tag consistent sentence set, Indicates that PN in the tth iteration t The matched and true label is the same as r i .tag consistent sentence set;

[0023] The rule redundancy is calculated as follows:

[0024]

[0025] in, Indicates that in the tth iteration, rule r i The set of sentences that are repeatedly matched by rules with the same tag. The higher the redundancy of a rule, the less useful information the rule contains.

[0026] The label is calculated as:

[0027]

[0028] When r i When .tag=1, the rule is considered a positive rule and is recorded as At this time, the rule matches the positive example; when r i When .tag=-1, the rule is considered a negative rule and counted as At this time, the rule matches the negative example; when r i When .tag=0, the rule is considered invalid.

[0029] In one embodiment, the divided training set is subjected to rule mining using a frequent pattern mining algorithm, including:

[0030] In the training set, the subject entity e is masked by the entity masking method. sub and object entity e obj Replace them with their corresponding entity types and use the frequent sequence mining algorithm to generate the rule set PN t , calculate the importance scores of the rules contained in the rule set based on the training set.

[0031] In one embodiment, in step S3, pattern matching is performed on sentences in a preset corpus using the converted rules, including:

[0032] Match each sentence in the preset corpus with each rule in the rule set to determine whether the subject entity type of the sentence is the same as the subject entity type of the rule, whether the object entity type of the sentence is the same as the object entity type of the rule, and whether the context word sequence in the sentence where the subject entity and object entity of the rule are located is a subsequence of the context word sequence in the sentence where the subject entity and object entity of the sentence are located. If all three are met, it means that the sentence successfully matches the rule pattern in the rule set. The formal expression of the judgment condition is s j R i Pattern matching is successful, s j is the jth sentence in the preset corpus, r i is the i-th rule in the rule set.

[0033] In one embodiment, the word-sentence semantic association graph G(V,E) in S4 is composed of a semantic association matrix A, represented by A∈R |V|×|V| , a mn Represents node v m and node v n The semantic association between m ,v n ∈V,a ij The value is the inclusion relationship or semantic similarity between two nodes, and is calculated as follows:

[0034]

[0035] If v m ,v n ∈W * or v m ,v n ∈S * , use the sim(m,n) function to calculate the semantic similarity between the two;

[0036] If v m ∈W * And v n ∈S * , use contain(m,n) to express the inclusion relationship between the two. When v n Contains v m When , contain(m,n)=1; otherwise, contain(m,n)=0;

[0037] The transition probability matrix is M=(P mn ) m,n∈V , where P mn Indicates that in the next step, from node v m Random walk to node v n Probability of:

[0038]

[0039] Given a node v m The random walker starts with the following rule:

[0040] P m (μ)=M T P m (μ-1)

[0041] Among them, P m (μ) indicates that it takes μ steps from the starting node to reach v m The probability of the node;

[0042] The random walk algorithm guided by semantic similarity probability is used to calculate the node v mand node v n The semantic matching score between is:

[0043]

[0044] Among them, P mn (μ) indicates that the random walker has walked μ steps from node v m Go to node v n The probability of k m Used to configure node v m The initial resources, |E| represents the number of edges in the graph; v m For words, v n is a sentence, then Sem mn (μ) represents the probability of walking from a word node to a sentence node after μ steps, m and n represent the node number corresponding to the word and the node number corresponding to the sentence, respectively;

[0045] The random walk algorithm based on semantic similarity probability guides the semantic matching of sentences in the preset corpus using rules, including:

[0046] Match each sentence in the preset corpus with each rule in the rule set, and judge whether the subject entity type of the sentence is the same as the subject entity type of the rule, whether the object entity type of the sentence is the same as the object entity type of the rule, and whether the semantic matching score between the node corresponding to the rule and the node corresponding to the sentence is greater than the threshold score. If all three are met, it means that the sentence successfully matches the rule pattern in the rule set. The formal expression of the judgment condition is s j .sub-type=r i .sub-type&s j .obj-type=r i .obj-type&Sem(r i ,s j )≥θ,s j R i Semantic matching is successful, s j is the jth sentence in the preset corpus, r i is the i-th rule in the rule set.

[0047] In one embodiment, the conflict resolution mechanism adopted in step S5 is:

[0048]

[0049] Among them, PN(s j ) represents the set of rules that successfully match the sentence pattern and semantics, including positive rules and negative rules, |PN(s j )| indicates a successful match jThe number of rule sets, I t (r i ) represents the rule r i The importance score in the tth iteration; Sem(r i ,s j ) represents the rule r i and sentence s j The matching score of When j For example; when When j is a negative example; when When j The label in this iteration is invalid;

[0050] Through the conflict resolution mechanism, the labels of the successfully matched sentences are determined and the target matching sentence set S is constructed. t .

[0051] In one embodiment, the method further comprises: using the trained machine learning models PC, SC and human resources, i.e., the manual labeling task volume H, to simultaneously match the sentence set S t Each sentence label of the training set S′ is predicted and verified, and the prediction result and confidence are output; there are two cases. Case 1: When the prediction results of the trained model PC and the trained SC are consistent, and the confidence of the two models is higher than the confidence threshold, the sentence label is directly output to the training set S′ t ; Case 2: When the prediction results of the trained model PC and the trained SC are inconsistent, and the confidence of the two models is higher than the confidence threshold, if H>0, then the sentence is assigned to manual annotation and the annotation results are output to the training set S′ t ; If H = 0, then return this type of sentence to S t , waiting for the next round of model prediction;

[0052] The trained models PC and SC use S′ t Retraining, S t Re-predict the sentences that are not output in until they match the sentence set S t If it is empty, the confidence of the prediction results of the trained models PC and SC are both lower than the confidence threshold; when the true label of the sentence is lacking, the collaborative prediction results of PC and SC are used as the true label of the sentence.

[0053] Based on the same inventive concept, the second aspect of the present invention provides a data annotation rule discovery device for entity relationship extraction, comprising:

[0054] A data set partitioning module is used to partition a preset seed data set into a training set and a test set;

[0055] The rule set construction module is used to define PN rules. Based on the defined PN rules, the frequent pattern mining algorithm is used to mine the divided training set and construct the rule set. The PN rules are composed of body, tag and I t (r i ) constitutes a mapping: r i .body→(r i .tag,I t (r i ), the rule body is a text pattern sequence of p = [w,@sub-type,w,@obj-type,w], where sub-type and obj-type represent the entity types of the subject and object respectively, w represents the context word sequence in the sentence where the subject entity and object entity are located, the tag represents the positive or negative of the rule, and the importance score I t (r i ) is used to determine the positive or negative nature of the rule;

[0056] The pattern matching module is used to convert the rules contained in the rule set into regular expressions, and use the converted rules to perform pattern matching on sentences in the preset corpus to obtain sentences that successfully match the rule pattern;

[0057] The semantic matching module is used to construct a word-sentence semantic association graph G(V,E) and a transition probability matrix M based on the rule set, and to semantically match the rules with sentences in the preset corpus based on a random walk algorithm guided by semantic similarity probability to obtain sentences that successfully match the semantics of the rules. The word-sentence semantic association graph G(V,E) is an undirected simple graph, where V represents the set of nodes, E represents the set of edges, E is a subset of V×V, and the node set V=W * ∪ * , W * represents the word set generated from the rules and candidate sentences, S * The candidate sentence set consists of sentences with the same subject and object entity types. The elements in the transition probability matrix are the probabilities of randomly walking from one node to another:

[0058] The conflict resolution module is used to resolve conflicts using a conflict resolution mechanism based on the sentences that successfully match the rule pattern and the sentences that successfully match the rule semantics, and to construct a target matching sentence set;

[0059] A collaborative training module is used to collaboratively train two machine learning models (PC and SC) based on the training set using a collaborative training method to obtain two trained classification models. The two trained classification models are then used to predict sentences in the target matching sentence set to obtain prediction results, which are then added to the training set. The prediction results are the pattern features learned by the machine learning model PC and the semantic features learned by the machine learning model SC.

[0060] The annotation result output module is used to repeat the operations of the rule set construction module, pattern matching module, semantic matching module, conflict resolution module and collaborative training module. When the frequent pattern mining algorithm cannot mine new rules, the iteration is stopped and the rule set and target matching sentence set are output. Among them, the sentences contained in the target matching sentence set and their corresponding labels are the data annotation results.

[0061] Based on the same inventive concept, the third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the method described in the first aspect when the program is executed.

[0062] Based on the same inventive concept, the fourth aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the program.

[0063] Compared with the prior art, the advantages and beneficial technical effects of the present invention are as follows:

[0064] The present application provides a data annotation rule discovery method and device for entity relationship extraction, wherein the method defines data annotation rules (PN rules) to represent positive rules and negative rules and their importance scores. PN rules can be used to jointly filter out incorrectly matched sentences to improve the accuracy of rule matching; a rule semantic matching mechanism based on a word-sentence association graph is designed, and a random walk algorithm guided by semantic similarity probability is used according to the semantic association between rules, words and sentences to improve the rule matching recall rate. The matching conflicts of PN rules will be resolved by a weighted voting strategy; a collaborative training rule verification mechanism is proposed to further verify the sentences matched by the rules, thereby improving the quality of iterative rule discovery. The present invention can solve the problem of overfitting of the entity relationship extraction model caused by insufficient training data, and iteratively generate data annotation rules under limited prior knowledge to improve the quality of data annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0066] Figure 1 A flowchart of the data annotation rule discovery method for entity relationship extraction provided by the present invention;

[0067] Figure 2 An example of a sentence matching positive and negative rules provided in an embodiment of the present invention;

[0068] Figure 3 An example of a rule semantic matching mechanism based on a word-sentence semantic association graph provided by an embodiment of the present invention;

[0069] Figure 4 An example of feature input for a rule verification model provided by an embodiment of the present invention;

[0070] Figure 5 An example of collaborative training of a rule verification model provided by an embodiment of the present invention;

[0071] Figure 6 A schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of the present invention;

[0072] Figure 7 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0073] This paper aims to solve the overfitting problem of entity relationship extraction models caused by insufficient training data, discover high-quality data annotation rules, improve the accuracy and recall of rule matching, and thus improve the quality of data annotation. This paper faces the following challenges:

[0074] The problem of low rule quality caused by mismatches has not been well addressed. Negative rules can be used to match negative examples, but conflicts can occur when positive and negative rules match the same sentence. Compared to pattern matching, semantic matching helps match similar sentences, but it can introduce additional errors. Based on limited human-labeled corpus, it is difficult to verify that rules match sentences, which affects the quality of new rules.

[0075] To address the above challenges, this paper proposes a data annotation rule discovery method for entity relationship extraction. The innovations are as follows:

[0076] First, we define PN rules to represent positive and negative rules and their importance scores. PN rules can be used to jointly filter out incorrectly matched sentences and improve the accuracy of rule matching.

[0077] Second, we designed a rule semantic matching mechanism based on the word-sentence association graph. Based on the semantic associations between rules, words, and sentences, we used a random walk algorithm guided by semantic similarity probability to improve the rule matching recall rate. PN rule matching conflicts were resolved using a weighted voting strategy.

[0078] Third, a collaborative training rule verification method is proposed to further verify the sentence matching of the rule, thereby improving the quality of iterative rule discovery.

[0079] This invention overcomes the low matching precision and recall of automatic rule discovery in existing technologies. It provides a data annotation rule discovery method for entity relationship extraction that can accurately identify positive and negative examples from unlabeled sentences, improving the accuracy and recall of rule matching and enhancing the quality of data annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0082] Example 1

[0083] An embodiment of the present invention provides a data annotation rule discovery method for entity relationship extraction, comprising:

[0084] S1: Divide the preset seed dataset into training set and test set;

[0085] S2: Define PN rules. Based on the defined PN rules, use the frequent pattern mining algorithm to mine the divided training set and build a rule set. The PN rules are composed of body, tag and I t (r i ) constitutes a mapping: r i .body→(r i .tag,I t (r i), the rule body is a text pattern sequence of p = [w,@sub-type,w,@obj-type,w], where sub-type and obj-type represent the entity types of the subject and object respectively, w represents the context word sequence in the sentence where the subject entity and object entity are located, the tag represents the positive or negative of the rule, and the importance score I t (r i ) is used to determine the positive or negative nature of the rule;

[0086] S3: Convert the rules contained in the rule set into regular expressions, use the converted rules to perform pattern matching on sentences in the preset corpus, and obtain sentences that successfully match the rule pattern;

[0087] S4: Based on the rule set, construct a word-sentence semantic association graph G(V,E) and a transition probability matrix M, and semantically match the rules to the sentences in the preset corpus based on the random walk algorithm guided by semantic similarity probability to obtain sentences that successfully match the semantics of the rules. The word-sentence semantic association graph G(V,E) is an undirected simple graph, V represents the set of nodes, E represents the set of edges, E is a subset of V×V, and the node set V=W * ∪ * , W * represents the word set generated from the rules and candidate sentences, S * The candidate sentence set consists of sentences with the same subject and object entity types. The elements in the transition probability matrix are the probabilities of randomly walking from one node to another:

[0088] S5: Based on the sentences that successfully match the rule pattern and the sentences that successfully match the rule semantics, the conflict resolution mechanism is used to resolve conflicts and construct the target matching sentence set;

[0089] S6: Based on the training set, two machine learning models (PC and SC) are collaboratively trained using a collaborative training method to obtain two trained classification models. The two trained classification models are then used to predict sentences in the target matching sentence set to obtain prediction results, which are added to the training set. The prediction results are the pattern features learned by the machine learning model PC and the semantic features learned by the machine learning model SC.

[0090] S7: Repeat steps S2 to S6. When the frequent pattern mining algorithm cannot mine new rules, stop the iteration and output the rule set and the target matching sentence set. The sentences contained in the target matching sentence set and their corresponding labels are the data annotation results.

[0091] See Figure 1 , which is a flowchart of the data annotation rule discovery method for entity relationship extraction provided by the present invention.

[0092] Specifically, the size of a small amount of data can be determined based on actual conditions, such as selecting 10%, 20%, etc. Importance score I t (r i ) In addition to determining the positive or negative nature of the rule, it is also used as a weight for voting when multiple rules successfully match the same sentence at the same time, which is the conflict resolution mechanism of step S5.

[0093] Compared to existing methods, the data annotation rule discovery method for entity relationship extraction disclosed in this paper is a quality-oriented, bootstrapped rule discovery method. Bootstrapping refers to gradually expanding the rule set based on a small number of seed rules through multiple rounds of iteration, and rule quality refers to the precision and recall of rule matching. This method has the following beneficial effects:

[0094] This method achieves a balance between rule matching precision and recall by expanding the sentences matched by the rules and then filtering out errors in these matched sentences, while also enhancing the interpretability of semantic matching.

[0095] This method reveals the potential of building high-quality rule sets using a quality-oriented bootstrapping mechanism. Since this method uses limited prior knowledge, it can be applied to highly specialized domains.

[0096] In addition, the rules discovered using this method can not only be used for data annotation, but also reveal knowledge associations, discover semantic regularities and explain text features.

[0097] In one embodiment, the importance score of the PN rule in step S2 is calculated as follows:

[0098]

[0099] Where σ is the normalization parameter, is the reliability of the rule, is the regular frequency, is the redundancy of rules;

[0100] The reliability of a rule is calculated as:

[0101]

[0102] Representation rule r i In the tth iteration, the sentences whose true labels are matched are positive. Representation rule r i The sentences whose true labels matched in the tth iteration are negative; and Representation rule r iThe sentence set obtained by pattern matching and semantic matching, when When , the rule is a positive rule; when When , the rule is a negative rule. This means that the rule is invalid in round t;

[0103] The rule frequency is calculated as:

[0104]

[0105] and Indicates that r in the tth iteration i Pattern matching and semantic matching and the actual label is the same as r i .tag consistent sentence set, Indicates that PN in the tth iteration t The matched and true label is the same as r i .tag consistent sentence set;

[0106] The rule redundancy is calculated as follows:

[0107]

[0108] in, Indicates that in the tth iteration, rule r i The set of sentences that are repeatedly matched by rules with the same tag. The higher the redundancy of a rule, the less useful information the rule contains.

[0109] The label is calculated as:

[0110]

[0111] When r i When .tag=1, the rule is considered a positive rule and is recorded as At this time, the rule matches the positive example; when r i When .tag=-1, the rule is considered a negative rule and counted as At this time, the rule matches the negative example; when r i When .tag=0, the rule is considered invalid.

[0112] See Figure 2 , which is an example of a sentence matching positive and negative rules provided by an embodiment of the present invention.

[0113] In one embodiment, the divided training set is subjected to rule mining using a frequent pattern mining algorithm, including:

[0114] In the training set, the subject entity e is masked by the entity masking method. sub and object entity e objReplace them with their corresponding entity types and use the frequent sequence mining algorithm to generate the rule set PN t , calculate the importance scores of the rules contained in the rule set based on the training set.

[0115] In one embodiment, in step S3, pattern matching is performed on sentences in a preset corpus using the converted rules, including:

[0116] Match each sentence in the preset corpus with each rule in the rule set to determine whether the subject entity type of the sentence is the same as the subject entity type of the rule, whether the object entity type of the sentence is the same as the object entity type of the rule, and whether the context word sequence in the sentence where the subject entity and object entity of the rule are located is a subsequence of the context word sequence in the sentence where the subject entity and object entity of the sentence are located. If all three are met, it means that the sentence successfully matches the rule pattern in the rule set. The formal expression of the judgment condition is s j R i Pattern matching is successful, s j is the jth sentence in the preset corpus, r i is the i-th rule in the rule set.

[0117] Specifically, when pattern matching a sentence with rules in a rule set, the rules are first converted into regular expressions, and then the sentence is matched with each rule.

[0118] In one embodiment, the word-sentence semantic association graph G(V,E) in S4 is composed of a semantic association matrix A, represented by A∈R |V|×|V| , a mn Represents node v m and node v n The semantic association between (m and n represent the node numbers respectively), where v m ,v n ∈V,a ij The value is the inclusion relationship or semantic similarity between two nodes, and is calculated as follows:

[0119]

[0120] If v m ,v n ∈W * or v m ,v n ∈S * , use the sim(m,n) function to calculate the semantic similarity between the two;

[0121] If v m ∈W* And v n ∈S * , use contain(m,n) to express the inclusion relationship between the two. When v n Contains v m When , contain(m,n)=1; otherwise, contain(m,n)=0;

[0122] The transition probability matrix is M=(P mn ) m,n∈V , where P mn Indicates that in the next step, from node v m Random walk to node v n Probability of:

[0123]

[0124] Given a node v m The random walker starts with the following rule:

[0125] P m (μ)=M T P m (μ-1)

[0126] Among them, P m (μ) indicates that it takes μ steps from the starting node to reach v m The probability of the node;

[0127] The random walk algorithm guided by semantic similarity probability is used to calculate the node v m and node v n The semantic matching score between is:

[0128]

[0129] Among them, P mn (μ) indicates that the random walker has walked μ steps from node v m Go to node v n The probability of k m Used to configure node v m The initial resources, |E| represents the number of edges in the graph; v m For words, v n is a sentence, then Sem mn (μ) represents the probability of walking from a word node to a sentence node after μ steps;

[0130] The random walk algorithm based on semantic similarity probability guides the semantic matching of sentences in the preset corpus using rules, including:

[0131] Match each sentence in the preset corpus with each rule in the rule set, and judge whether the subject entity type of the sentence is the same as the subject entity type of the rule, whether the object entity type of the sentence is the same as the object entity type of the rule, and whether the semantic matching score between the node corresponding to the rule and the node corresponding to the sentence is greater than the threshold score. If all three are met, it means that the sentence successfully matches the rule pattern in the rule set. The formal expression of the judgment condition is s j .sub-type=r i .sub-type&s j .obj-type=r i .obj-type&Sem(r i ,s j )≥θ,s j R i Semantic matching is successful, s j is the jth sentence in the preset corpus, r i is the i-th rule in the rule set.

[0132] See Figure 3 , which is an example of a rule semantic matching mechanism based on a word-sentence semantic association graph provided by an embodiment of the present invention.

[0133] In one embodiment, the conflict resolution mechanism adopted in step S5 is:

[0134]

[0135] Among them, PN(s j ) represents the set of rules that successfully match the sentence pattern and semantics, including positive rules and negative rules, |PN(s j )| indicates a successful match j The number of rule sets, I t (r i ) represents the rule r i Importance score in the tth iteration; Representation rule r i and sentence s j The matching score of When j For example; when When j is a negative example; when When j The label in this iteration is invalid;

[0136] Through the conflict resolution mechanism, the labels of the successfully matched sentences are determined and the target matching sentence set S is constructed. t .

[0137] In one embodiment, the method further comprises: using the trained machine learning models PC, SC and human resources, i.e., the manual labeling task volume H, to simultaneously match the sentence set S t Each sentence label of the training set S′ is predicted and verified, and the prediction result and confidence are output; there are two cases. Case 1: When the prediction results of the trained model PC and the trained SC are consistent, and the confidence of the two models is higher than the confidence threshold, the sentence label is directly output to the training set S′ t ; Case 2: When the prediction results of the trained model PC and the trained SC are inconsistent, and the confidence of the two models is higher than the confidence threshold, if H>0, then the sentence is assigned to manual annotation and the annotation results are output to the training set S′ t ; If H = 0, then return this type of sentence to S t , waiting for the next round of model prediction;

[0138] The trained models PC and SC use S′ t Retraining, S t Re-predict the sentences that are not output in until they match the sentence set S t If it is empty, the confidence of the prediction results of the trained models PC and SC are both lower than the confidence threshold; when the true label of the sentence is lacking, the collaborative prediction results of PC and SC are used as the true label of the sentence.

[0139] See Figure 4 and Figure 5 ,in, Figure 4 An example of feature input for a rule verification model provided by an embodiment of the present invention; Figure 5 This is an example of collaborative training of a rule verification model provided by one embodiment of the present invention.

[0140] In a specific application process, the overall algorithm involved in the present invention is as follows:

[0141]

[0142] The specific embodiments are as follows:

[0143] We construct a friendship corpus from the IPRE dataset, which consists of 5,000 sentences, and a seed dataset, which consists of 500 sentences. We divide the seed dataset sentences into a training set and a test set in a ratio of 7:3. Based on the training set, we can obtain PN rules through sentence preprocessing and rule mining algorithm. and like Figure 2 As shown, its initial importance score is 1. Then, in the tth round of iteration, based on the mined PN rules and The present invention will perform rule pattern matching and semantic matching. Figure 2 As shown, Sentences s1, s2, s3, and s4 are matched successfully. Sentence s3 is successfully matched. Among them, s2 and s4 are If the semantic matching is successful, the specific process of the semantic matching can be seen Figure 3 :In the word-sentence semantic association graph, due to Contains the word "together", and the semantic similarity between the words "together" (w1) and "together" (w2) is 0.8. Since sentence s2 contains the word "together", the probability of jumping from w1 to s2, i.e. the semantic matching score, is 0.8, which is higher than the semantic matching threshold θ = 0.65. S2 can be Semantic matching is successful. Similarly, s4 can be The semantic matching is successful, and its semantic matching path is: w1→w2→s2→s4. The semantic matching score is 0.72, which is higher than the semantic matching threshold. Then, using the training set, the present invention will train two support vector machine models (PC, SC), and use the trained models to predict the matching sentence set. Among them, the two models predict the label of s3 as -1, and the confidence of the prediction is higher than 0.7. Therefore, the prediction result of s3 as a negative example is input into the training set. Similarly, the model predicts that the labels of s1 and s2 are 1, and the confidence of the prediction is higher than 0.7. The present invention inputs the prediction results of s1 and s2 as positive examples into the training set. In addition, the two models have inconsistent prediction labels for s4. Therefore, the present invention uses manual annotation of s4 as a positive example and inputs it into the training set. So far, the matching sentence set has been fully verified. Based on the training set, and Perform importance assessment and update its importance score. Figure 1 Rule importance assessment part, due to The match was successful for three positive examples and one negative example, so its reliability score is Frequency score Redundancy score Therefore, its importance score Similarly, Reliability score Frequency score Redundancy score Therefore, its importance score According to the importance score, is a positive rule, In the t+1th iteration, based on s1 to s4, the present invention mines a new rule Its initial importance score is 0.2, which is a positive rule. In the rule matching phase, s5 is matched successfully again. Since s3 is matched successfully by these three rules at the same time, and the types of these rules are inconsistent, a rule matching conflict occurs. The present invention uses the rule conflict resolution mechanism to calculate the label of s3. Therefore, the rule label of s3 is -1. This also proves that the positive and negative rules and rule conflict resolution mechanism can effectively filter out incorrectly labeled sentences and improve the accuracy of rule matching. Figure 4 In the rule verification phase, feature learning is performed based on the training set. Since the vector dimensions of the pattern features are different, the total feature vector is obtained by splicing and input into the classification model PC. Since the vector dimensions of the semantic features are the same, in order to obtain the overall semantics of the sentence, the sentence vector is obtained by averaging the word vectors and input into the classification model SC. The threshold α is set to 0.7. Since the prediction results of s5 by the PC and SC models are different and the confidence level is higher than the threshold, the present invention uses manual annotation to obtain the sentence as a negative example (such as Figure 5 ).therefore, Reliability score Frequency score Redundancy score Therefore, its importance score because and Both are positive rules, and they successfully match the same positive example, so The redundancy score increases, resulting in The importance score of the rule is reduced to 0.2. Through the bootstrapped iterative process of rule mining, rule matching, and rule verification, the present invention gradually builds a high-quality rule set, improves the accuracy and recall rate of rule matching, and outputs the rule annotation results, i.e., the data annotation results, until the rule set becomes stable.

[0144] The data annotation rule discovery method for entity relationship extraction provided by this invention can perform bootstrapped rule discovery from unstructured text, improve the rule matching recall rate using a semantic matching mechanism, and improve the rule matching precision rate using positive and negative rules and rule verification mechanisms, thereby achieving high-quality data annotation rule discovery. The final data annotation results can be used to improve the quality and effectiveness of entity relationship extraction.

[0145] Example 2

[0146] Based on the same inventive concept, this embodiment provides a data annotation rule discovery device for entity relationship extraction, comprising:

[0147] A data set partitioning module is used to partition a preset seed data set into a training set and a test set;

[0148] The rule set construction module is used to define PN rules. Based on the defined PN rules, the frequent pattern mining algorithm is used to mine the divided training set and construct the rule set. The PN rules are composed of body, tag and I t (r i ) constitutes a mapping: r i .body→(r i .tag,I t (r i ), the rule body is a text pattern sequence of p = [w,@sub-type,w,@obj-type,w], where sub-type and obj-type represent the entity types of the subject and object respectively, w represents the context word sequence in the sentence where the subject entity and object entity are located, the tag represents the positive or negative of the rule, and the importance score I t (r i ) is used to determine the positive or negative nature of the rule;

[0149] The pattern matching module is used to convert the rules contained in the rule set into regular expressions, and use the converted rules to perform pattern matching on sentences in the preset corpus to obtain sentences that successfully match the rule pattern;

[0150] The semantic matching module is used to construct a word-sentence semantic association graph G(V,E) and a transition probability matrix M based on the rule set, and to semantically match the rules with sentences in the preset corpus based on a random walk algorithm guided by semantic similarity probability to obtain sentences that successfully match the semantics of the rules. The word-sentence semantic association graph G(V,E) is an undirected simple graph, where V represents the set of nodes, E represents the set of edges, E is a subset of V×V, and the node set V=W * ∪ * , W * represents the word set generated from the rules and candidate sentences, S * The candidate sentence set consists of sentences with the same subject and object entity types. The elements in the transition probability matrix are the probabilities of randomly walking from one node to another:

[0151] The conflict resolution module is used to resolve conflicts using a conflict resolution mechanism based on the sentences that successfully match the rule pattern and the sentences that successfully match the rule semantics, and to construct a target matching sentence set;

[0152] A collaborative training module is used to collaboratively train two machine learning models (PC and SC) based on the training set using a collaborative training method to obtain two trained classification models. The two trained classification models are then used to predict sentences in the target matching sentence set to obtain prediction results, which are then added to the training set. The prediction results are the pattern features learned by the machine learning model PC and the semantic features learned by the machine learning model SC.

[0153] The annotation output module is used to repeat the operations of the rule set construction module, pattern matching module, semantic matching module, conflict resolution module and collaborative training module. When the frequent pattern mining algorithm cannot mine new rules, the iteration is stopped and the rule set and target matching sentence set are output. Among them, the sentences contained in the target matching sentence set and their corresponding labels are the data annotation results.

[0154] Since the device described in the second embodiment of the present invention is the device used to implement the data annotation rule discovery method for entity relationship extraction in the first embodiment of the present invention, based on the method described in the first embodiment of the present invention, those skilled in the art can understand the specific structure and variations of the device, so they are not described in detail here. All devices used in the method in the first embodiment of the present invention fall within the scope of protection of the present invention.

[0155] Example 3

[0156] Based on the same inventive concept, see Figure 6 The present invention further provides a computer-readable storage medium 300 on which a computer program 311 is stored. When the program is executed, the method described in the first embodiment is implemented.

[0157] Since the computer-readable storage medium described in the third embodiment of the present invention is the computer-readable storage medium used to implement the data annotation rule discovery method for entity relationship extraction in the first embodiment of the present invention, those skilled in the art will be able to understand the specific structure and variations of the computer-readable storage medium based on the method described in the first embodiment of the present invention, and therefore will not be described in detail here. All computer-readable storage media used in the method of the first embodiment of the present invention fall within the scope of protection of the present invention.

[0158] Example 4

[0159] Based on the same inventive concept, the present application also provides a computer device, such as Figure 7 As shown, it includes a memory 401, a processor 402 and a computer program 403 stored in the memory and executable on the processor. When the processor executes the above program, the method in the first embodiment is implemented.

[0160] Since the computer device introduced in the fourth embodiment of the present invention is the computer device used to implement the data annotation rule discovery method for entity relationship extraction in the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, those skilled in the art can understand the specific structure and variations of the computer device, so they are not described in detail here. All computer devices used in the method in the first embodiment of the present invention fall within the scope of protection of the present invention.

[0161] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0162] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0163] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0164] Obviously, those skilled in the art may make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if such changes and modifications of the embodiments of the present invention fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A data annotation rule discovery method for entity relationship extraction, characterized by: include: S1: Divide the preset seed dataset into training set and test set; S2: Define PN rules. Based on the defined PN rules, use the frequent pattern mining algorithm to mine the divided training set and build a rule set. The PN rules are composed of body, tag and I t (r i ) constitutes a mapping: r i .body→(r i .tag,I t (r i ), the rule body is a text pattern sequence of p = [w,@sub-type,w,@obj-type,w], where sub-type and obj-type represent the entity types of the subject and object respectively, w represents the context word sequence in the sentence where the subject entity and object entity are located, the tag represents the positive or negative of the rule, and the importance score I t (r i ) is used to determine the positive or negative nature of the rule; S3: Convert the rules contained in the rule set into regular expressions, use the converted rules to perform pattern matching on sentences in the preset corpus, and obtain sentences that successfully match the rule pattern; S4: Based on the rule set, construct a word-sentence semantic association graph G(V,E) and a transition probability matrix M, and semantically match the rules to the sentences in the preset corpus based on the random walk algorithm guided by semantic similarity probability to obtain sentences that successfully match the semantics of the rules. The word-sentence semantic association graph G(V,E) is an undirected simple graph, V represents the set of nodes, E represents the set of edges, E is a subset of V×V, and the node set V=W * ∪S * , W * represents the word set generated from the rules and candidate sentences, S * The candidate sentence set consists of sentences with the same subject and object entity types. The elements in the transition probability matrix are the probabilities of randomly walking from one node to another: S5: Based on the sentences that successfully match the rule pattern and the sentences that successfully match the rule semantics, the conflict resolution mechanism is used to resolve conflicts and construct the target matching sentence set; S6: Based on the training set, two machine learning models (PC and SC) are collaboratively trained using a collaborative training method to obtain two trained classification models. The two trained classification models are then used to predict sentences in the target matching sentence set to obtain prediction results, which are added to the training set. The prediction results are the pattern features learned by the machine learning model PC and the semantic features learned by the machine learning model SC. S7: Repeat steps S2 to S6. When the frequent pattern mining algorithm cannot mine new rules, stop iteration and output the rule set and the target matching sentence set. The sentences contained in the target matching sentence set and their corresponding labels are the data annotation results. The data annotation results can be used as the training set of the entity relationship extraction model.

2. The data annotation rule discovery method for entity relationship extraction according to claim 1, characterized in that: The importance score of the PN rule in step S2 is calculated as follows: Where σ is the normalization parameter, is the reliability of the rule, is the regular frequency, is the redundancy of rules; The reliability of a rule is calculated as: Representation rule r i In the tth iteration, the sentences whose true labels are matched are positive. Representation rule r i The sentences whose true labels matched in the tth iteration are negative; and Representation rule r i The sentence set obtained by pattern matching and semantic matching, when When , the rule is a positive rule; when When , the rule is a negative rule. This means that the rule is invalid in round t; The rule frequency is calculated as: and Indicates that r in the tth iteration i Pattern matching and semantic matching and the actual label is the same as r i .tag consistent sentence set, Indicates that PN in the tth iteration t The matched and true label is the same as r i .tag consistent sentence set; The rule redundancy is calculated as follows: in, Indicates that in the tth iteration, rule r i The set of sentences that are repeatedly matched by rules with the same tag. The higher the redundancy of a rule, the less useful information the rule contains. The label is calculated as: When r i When .tag=1, the rule is considered a positive rule and is recorded as At this time, the rule matches the positive example; when r i When .tag=-1, the rule is considered a negative rule and counted as At this time, the rule matches the negative example; when r i When .tag=0, the rule is considered invalid.

3. The data annotation rule discovery method for entity relationship extraction according to claim 1, characterized in that: The divided training set is subjected to rule mining using a frequent pattern mining algorithm, including: In the training set, the subject entity e is masked by the entity masking method. sub and object entity e obj Replace them with their corresponding entity types and use the frequent sequence mining algorithm to generate the rule set PN t , calculate the importance scores of the rules contained in the rule set based on the training set.

4. The data annotation rule discovery method for entity relationship extraction according to claim 1, characterized in that: In step S3, the converted rules are used to perform pattern matching on sentences in the preset corpus, including: Match each sentence in the preset corpus with each rule in the rule set, and judge whether the subject entity type of the sentence is the same as the subject entity type of the rule, whether the object entity type of the sentence is the same as the object entity type of the rule, and whether the context word sequence in the sentence where the subject entity and object entity of the rule are located is a subsequence of the context word sequence in the sentence where the subject entity and object entity of the sentence are located. If all three are met, it means that the sentence successfully matches the rule pattern in the rule set. The formal expression of the judgment condition is s j .sub-type=r i .sub-type&s j .obj-type=r i .obj-type& s j R i Pattern matching is successful, s j is the jth sentence in the preset corpus, r i is the i-th rule in the rule set.

5. The data annotation rule discovery method for entity relationship extraction according to claim 1, characterized in that: The word-sentence semantic association graph G(V,E) in S4 is composed of the semantic association matrix A, represented by A∈R |V|×|V| , a mn Represents node v m and node v n The semantic association between m ,v n ∈V,a ij The value is the inclusion relationship or semantic similarity between two nodes, and is calculated as follows: If v m ,v n ∈W * or v m ,v n ∈S * , use the sim(m,n) function to calculate the semantic similarity between the two; If v m ∈W * And v n ∈S * , use contain(m,n) to express the inclusion relationship between the two. When v n Contains v m When contain(m,n)=1; Otherwise, contain(m,n)=0; The transition probability matrix is M=(P mn ) m,n∈V , where P mn Indicates that in the next step, from node v m Random walk to node v n Probability of: Given a node v m The random walker starts with the following rule: P m (μ)=M T P m (μ-1) Among them, P m (μ) indicates that it takes μ steps from the starting node to reach v m The probability of the node; The random walk algorithm guided by semantic similarity probability is used to calculate the node v m and node v n The semantic matching score between is: Among them, P mn (μ) indicates that the random walker has walked μ steps from node v m Go to node v n The probability of k m Used to configure node v m The initial resources, |E| represents the number of edges in the graph; v m For words, v n is a sentence, then Sem mn (μ) represents the probability of walking from a word node to a sentence node after μ steps, m and n represent the node number corresponding to the word and the node number corresponding to the sentence, respectively; The random walk algorithm based on semantic similarity probability guides the semantic matching of sentences in the preset corpus using rules, including: Match each sentence in the preset corpus with each rule in the rule set, and judge whether the subject entity type of the sentence is the same as the subject entity type of the rule, whether the object entity type of the sentence is the same as the object entity type of the rule, and whether the semantic matching score between the node corresponding to the rule and the node corresponding to the sentence is greater than the threshold score. If all three are met, it means that the sentence successfully matches the rule pattern in the rule set. The formal expression of the judgment condition is s j .sub-type=r i .sub-type&s j .obj-type=r i .obj-type&Sem(r i ,s j )≥θ,s j R i Semantic matching is successful, s j is the jth sentence in the preset corpus, r i is the i-th rule in the rule set.

6. The data annotation rule discovery method for entity relationship extraction according to claim 1, characterized in that: The conflict resolution mechanism adopted in step S5 is: Among them, PN(s j ) represents the set of rules that successfully match the sentence pattern and semantics, including positive rules and negative rules, |PN(s j )| indicates a successful match j The number of rule sets, I t (r i ) represents the rule r i The importance score in the tth iteration; Sem(r i ,s j ) represents the rule r i and sentence s j The matching score of When , it indicates that the sentence s j For example; when When , it indicates that the sentence s j is a negative example; when When j The label in this iteration is invalid; Through the conflict resolution mechanism, the labels of the successfully matched sentences are determined and the target matching sentence set S is constructed. t .

7. The data annotation rule discovery method for entity relationship extraction according to claim 1, characterized in that: The method further includes: using the trained machine learning models PC, SC and human resources, i.e., the manual labeling task volume H, to simultaneously match the sentence set S t Each sentence label of the training set S′ is predicted and verified, and the prediction result and confidence are output; there are two cases. Case 1: When the prediction results of the trained model PC and the trained SC are consistent, and the confidence of the two models is higher than the confidence threshold, the sentence label is directly output to the training set S′ t ; Case 2: When the prediction results of the trained model PC and the trained SC are inconsistent, and the confidence of the two models is higher than the confidence threshold, if H>0, then the sentence is assigned to manual annotation and the annotation results are output to the training set S′ t ; If H = 0, then return this type of sentence to S t , waiting for the next round of model prediction; The trained models PC and SC use S′ t Retraining, S t Re-predict the sentences that are not output in until they match the sentence set S t If it is empty, the confidence of the prediction results of the trained models PC and SC are both lower than the confidence threshold; when the true label of the sentence is lacking, the collaborative prediction results of PC and SC are used as the true label of the sentence.

8. A data annotation rule discovery device for entity relationship extraction, characterized in that: include: A data set partitioning module is used to partition a preset seed data set into a training set and a test set; The rule set construction module is used to define PN rules. Based on the defined PN rules, the frequent pattern mining algorithm is used to mine the divided training set and construct the rule set. The PN rules are composed of body, tag and I t (r i ) constitutes a mapping: r i .body→(r i .tag,I t (r i ), the rule body is a text pattern sequence of p = [w,@sub-type,w,@obj-type,w], where sub-type and obj-type represent the entity types of the subject and object respectively, w represents the context word sequence in the sentence where the subject entity and object entity are located, the tag tag represents the positive or negative of the rule, and the importance score i t (r i ) is used to determine the positive or negative nature of the rule; The pattern matching module is used to convert the rules contained in the rule set into regular expressions, and use the converted rules to perform pattern matching on sentences in the preset corpus to obtain sentences that successfully match the rule pattern; The semantic matching module is used to construct a word-sentence semantic association graph G(V,E) and a transition probability matrix M based on the rule set, and to semantically match the rules with sentences in the preset corpus based on a random walk algorithm guided by semantic similarity probability to obtain sentences that successfully match the semantics of the rules. The word-sentence semantic association graph G(V,E) is an undirected simple graph, where V represents the set of nodes, E represents the set of edges, E is a subset of V×V, and the node set V=W * ∪ * , W * represents the word set generated from the rules and candidate sentences, S * The candidate sentence set consists of sentences with the same subject and object entity types. The elements in the transition probability matrix are the probabilities of randomly walking from one node to another: The conflict resolution module is used to resolve conflicts using a conflict resolution mechanism based on the sentences that successfully match the rule pattern and the sentences that successfully match the rule semantics, and to construct a target matching sentence set; A collaborative training module is used to collaboratively train two machine learning models (PC and SC) based on the training set using a collaborative training method to obtain two trained classification models. The two trained classification models are then used to predict sentences in the target matching sentence set to obtain prediction results, which are then added to the training set. The prediction results are the pattern features learned by the machine learning model PC and the semantic features learned by the machine learning model SC. The annotation result output module is used to repeat the operations of the rule set construction module, pattern matching module, semantic matching module, conflict resolution module and collaborative training module. When the frequent pattern mining algorithm cannot mine new rules, the iteration is stopped and the rule set and target matching sentence set are output. Among them, the sentences contained in the target matching sentence set and their corresponding labels are the data annotation results.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed, the method according to any one of claims 1 to 7 is implemented.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

Images