Weakly supervised and explainable learning of machine learning-based named entity recognition (NER) mechanisms

Abstract

To provide a system for and a method of making a machine learning model for use in executing named entity recognition with weakly supervised mechanism.SOLUTION: A method according to the present invention has a step of automatically discriminating an entity candidate and a regulation candidate in an input of an unlabeled text to use a label allocated to the entity candidate by a set of seeding rules in an initial training of a machine learning model as a first set of training data for execution. The method has a subsequent step of applying the trained machine learning model to the unlabeled text to discriminate, from the rule candidates, a subset of rules for generating a label most precisely matched with the label allocated by the trained machine learning model. The method further has a step of using the label allocated by the discriminated subset of the rules as a second set of the training data for re-training. These steps are repeatedly carried out.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] background The present invention relates to systems and methods for performing named entity recognition (NER) using machine learning techniques, and more particularly to systems and methods for training named entity recognition (NER) models. [Background technology]

[0002] overview Named entity recognition (NER) is a mechanism for applying automated processing (e.g., computer-based processing) to unstructured text to identify and categorize occurrences of "named entities" (e.g., people, businesses, places, etc.) in the text. For example, in some implementations, NER is a machine learning-based natural language processing mechanism in which an unstructured natural language sentence is provided as input to a machine learning model, and the output of the machine learning model includes an indication of the category assigned to each "entity" (or potential entity) in the sentence (e.g., that the machine learning model determines a word or phrase appearing in the sentence may correspond to an appropriate name, object, etc.). For example, if the input sentence provided as input states "John is traveling to London," the output of a trained NER machine learning model may indicate that "John" is categorized as a "person" and "London" is categorized as a "place." Summary of the Invention [Problem to be solved by the invention]

[0003] In some implementations, named entity recognition (NER) is a critical task for many downstream information extraction tasks (e.g., relation extraction) and knowledge base construction. Supervised learning of named entity recognition has achieved reliable performance, for example, due to advances in deep neural models. However, supervised learning of NER models requires manual annotation of large amounts of data for training. This can take a significant amount of time in any case, but is particularly problematic when training NER models in some specific domains and / or for low-resource languages ​​where domain expert annotation is difficult to obtain.

[0004] In some implementations, "distance-supervised" learning is used to automatically generate labeled data from an open knowledge base or dictionary. Distance-supervised methods allow for the large-scale generation of training data for NER models without expensive human labor. However, all distance-supervised methods rely on existing knowledge bases or dictionaries, and in some cases, open knowledge bases are not available (e.g., in the biomedical field, technical literature, etc.). [Means for solving the problem]

[0005] Thus, in some implementations, the systems and methods described herein provide a "weakly supervised" mechanism for training a machine learning NER model. In a weakly supervised approach, a small set of symbolic rules (referred to herein as "seeding rules") is used to label data in unstructured text. In some implementations, the seeding rules and their associated labels may be manually provided or defined for a particular task (i.e., the task for which the NER model is trained). After the seeding rules are applied to the unstructured text, the weakly labeled data is used to train an initial iteration of an artificial neural network-based NER model. The unstructured text is also processed to automatically identify multiple potential rules for labeling "named entities." The automatically identified rules are applied to the unstructured text, and the text / label combinations determined by the rules are compared to the text / label combinations determined by the initial iteration of the NER model. The most successful "rules" are identified using a scoring metric and then applied to the original unstructured text to generate another set of training data. The NER model is then retrained based on data labeled with the new set of selected rules, and this training process is repeated iteratively to continually refine and improve the NER model.

[0006] In some implementations, a "weakly supervised" mechanism for training an NER model uses bootstrapping to generate weakly labeled data with symbolic rules and automatically trains the NER model to recognize entities with neural representations. For example, in some implementations, initial seeding rules may include rules such as "located in" to explicitly identify at least some locations within unstructured text. Furthermore, the NER model can be trained to identify new entities by comparing low-dimensional neural representations (i.e., word embeddings) and iteratively retraining the NER model. The following examples describe a framework for iteratively discovering new entities from unlabeled corpora (e.g., unstructured text) using both explicit logical rules and neural representations. Furthermore, because the systems and methods use logical rules to obtain weak labels and recognize entities, each system prediction provided by a trained NER model can be traced back to the original logical rules, making the prediction results explainable.

[0007] In one embodiment, the present invention provides a method for training a machine learning model to perform named entity recognition. In an input dataset of unlabeled text, all possible entity candidates and all possible rule candidates are automatically identified. Initial training of the machine learning model is performed by applying a set of seeding rules to the input dataset to assign labels to the entity candidates and using the label assignments as a first set of training data. The trained machine learning model is then applied to the unlabeled text, and a subset of rules is identified from the rule candidates that create labels that most accurately match the labels assigned by the trained machine learning model. The machine learning model is then retrained using the labels assigned by the identified subset of rules as a second set of training data. The processes of applying the retrained model, identifying the subset of rules that assign labels that most accurately match the labels assigned by the retrained model, and performing additional retraining of the model are repeated iteratively to further refine and improve the performance of the machine learning model for named entity recognition.

[0008] According to another embodiment, the present invention provides a system for training a machine learning model to perform named entity recognition. The system includes an electronic processor configured to identify all possible entity candidates and all possible rule candidates in an input dataset of unlabeled text. The electronic processor performs initial training of the machine learning model by applying a set of seeding rules to the input dataset to assign labels to the entity candidates and using the label assignments as a first set of training data. The electronic processor then applies the trained machine learning model to the unlabeled text and identifies a subset of rules from the rule candidates that create labels that most accurately match the labels assigned by the trained machine learning model. The electronic processor then retrains the machine learning model using the labels assigned by the identified subset of rules as a second set of training data. The processes of applying the retrained model, identifying the subset of rules that assign labels that most accurately match the labels assigned by the retrained model, and performing additional retraining of the model are repeated iteratively to further refine and improve the performance of the machine learning model for named entity recognition.

[0009] Other aspects of the invention will become apparent by consideration of the detailed description and accompanying drawings. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram of a system for training and using a machine learning based named entity recognition (NER) mechanism, according to one embodiment. [Figure 2] FIG. 2 is a schematic diagram of one embodiment of a NER mechanism trained and / or applied using the system of FIG. 1. [Figure 3]FIG. 2 is a schematic diagram of a system framework for training the NER mechanism in the system of FIG. 1 using automatically generated logic rules. [Figure 4] 4 is a flowchart of a method for training the NER mechanism of FIG. 2 using the system framework of FIG. 3. DETAILED DESCRIPTION OF THE INVENTION

[0011] Detailed Description Before describing any embodiments of the present invention in detail, it is to be understood that the invention is not limited in its application to the details of construction and the arrangement of components set forth in the following description or illustrated in the accompanying drawings. The invention is capable of other embodiments and of being practiced or carried out in various ways.

[0012] 1 illustrates one embodiment of a computer-based system 100 that can be configured to train a named entity recognition (NER) machine learning mechanism, apply a trained NER mechanism, or both. System 100 includes an electronic processor 101 and non-transitory computer-readable memory 103. Memory 103 stores data and computer-executable instructions that are accessed and executed by electronic processor 101 to provide functionality of system 100, including, for example, functionality described herein below. Electronic processor 101 is communicatively coupled to a display 105 and user input devices 107 (e.g., keyboard, mouse, touchscreen, etc.) to provide a user interface for operating system 100 and displaying data to a user. Electronic processor 101 is also communicatively coupled to input / output devices 109 (e.g., wired or wireless communication interfaces) for communicating with other computer-based systems.

[0013] FIG. 2 illustrates an example of a machine learning model that may be trained and / or applied by the system 100 of FIG. 1. The machine learning model 201 is configured to receive as input unstructured and unlabeled text, including, for example, a text sentence 203. In response to receiving the input text, the machine learning model is configured to output each of a plurality of "spans" of text from the sentence and a label assigned to each span. The labels output by the machine learning model 201 indicate whether the span was identified as a named entity and, if so, what category label was assigned to the span. In the example of FIG. 2, the machine learning model has identified three distinct spans 205, 207, and 209 within the input sentence 203. The first span 205 has been assigned a "person" label 211 by the machine learning model 201, indicating that the text in the first span 205 has been identified as corresponding to a person's name. The second span 207 is assigned a "location" label 213 by the machine learning model 201, indicating that the text in the second span 207 was identified as corresponding to the name of a location (e.g., a building, a city, a state, a country, etc.). Finally, the third span 209 is assigned a "NEG" label 215 by the machine learning model 201. As described in more detail below, the "NEG" label 215 indicates that the machine learning model 201 determined that the text in the third span 209 does not correspond to any named entity.

[0014] Each "span" may include a single word from the input sentence or a combination of multiple words from the input sentence. For example, if the sentence "I like running" is provided as input to the machine learning model 201, the machine learning model in some implementations may be configured to produce the following spans as output: [I], [like], [running], [I like], [like running], and [I like running]. While the particular example of FIG. 2 shows the machine learning model 201 producing only three spans as output, in other implementations the machine learning model 201 may be configured to output more or fewer identified spans. Similarly, in some implementations, the number of spans produced as output may vary depending on the sentence provided as input. Indeed, in some implementations the machine learning model 201 may be configured to produce all possible spans in the input sentence as output.

[0015] As a further example, if the sentence “George lives in London” is provided as input 203, the machine learning model to be trained 201 may select the following combination of span and label: [Table 1] as an output.

[0016] Figure 3 illustrates one embodiment of a framework for training the machine learning model 201 of Figure 2 to perform named entity recognition and labeling on unstructured text input. Both an entity candidate generator 301 and a rule candidate generator 303 are applied to a set of unlabeled training data 305. The entity candidate generator 301 is configured to automatically process the input text 305 to identify all possible candidates (e.g., "spans") within the input text 305. Similarly, the rule candidate generator 303 is configured to automatically generate potential "candidate rules" from the unlabeled data that can be used to determine whether a particular "span" is a named entity.

[0017] The entity candidates and rule candidates are provided as input to an iterative NER training module 307. A rule labeler 309 automatically applies a set of labeling rules 311 to each entity candidate and assigns a label to the entity candidate. As described in more detail below, in the first iteration of the iterative NER training module 307, the labeling rules 311 include a base set of seeding rules 313. The labeled data from the rule labeler 309 is then provided as training input for a neural NER model 315. The original unlabeled data 305 is then provided as input data to the trained neural NER model 315, which produces a "predicted data" output. The predicted data includes the IDs of one or more spans and the labels assigned to the spans by the trained neural NER model 315 (e.g., see Table 1 above). The rule selector 317 is then configured to score and select the most accurate labeling rule from the set of rule candidates (generated by the rule candidate generator 303) by applying the rule candidates to the unlabeled data and comparing the results of each rule with the predicted data output by the neural NER model 315.

[0018] The set of rule candidates identified by the rule selector 317 as most accurate is then used as the labeling rules 311 for the next iteration. In the next iteration of the iterative NER training module 307, the rule labeler 309 applies the selected set of rules to the entity candidates to create a new set of labeled data, which is used as training data to retrain the neural NER model 315. The updated neural NER model 315 is then applied to the unlabeled data 305 to create a new set of predicted data, and the rule selector 317 identifies the set of rule candidates that produces results that most accurately match the output of the updated neural NER model 315. In various implementations, this iterative process 307 is repeated until a termination condition is reached (e.g., after a specified number of iterations, after a specified performance metric is achieved, or until the rule selector 317 converges on a particular set of labeling rules).

[0019] Figure 4 illustrates one embodiment of a method applied by the system of Figure 1 to train a machine learning mechanism (e.g., an artificial neural network) to perform named entity recognition and labeling (e.g., as shown in the embodiment of Figure 2) using the framework of Figure 3. The system 100 begins by applying the entity candidate generator 301 to the unlabeled data 305 to identify all entity candidates (step 401) and the rule candidate generator 303 to the unlabeled data to identify all rule candidates (step 403). The system 100 then applies a set of seeding rules 213 to the entity candidates to label them (step 405) and uses this "weakly labeled" data set as training data to train the neural NER model 315 (step 407). The system 100 then applies the initially trained neural NER model 315 to the original unlabeled data 305 (step 409) to create a set of "predicted data" labels. Each rule candidate from the rule candidate generator 303 is also applied to the original unlabeled data (step 411), and the accuracy of each rule candidate is scored by comparing the result of the rule candidate with the set of "predicted data" labels from the initially trained neural NER model 315 (step 413). The system identifies the best-performing rule candidate (e.g., the rule candidate that produces labels that most accurately match the set of "predicted data" labels produced by the neural NER model) (step 415). The system 100 then applies the identified set of best-performing rule candidates to the original unlabeled data as new labeling rules (step 417), and retrains the neural NER model using the data labeled with this new set of labeling rules as training data (step 407).The retrained neural NER model 315 is then applied to the original unlabeled data to generate a new set of "predicted data" labels (step 409), and each rule candidate is again scored (step 413), this time by comparing the labels generated by each rule candidate with the new set of predicted data labels generated by the retrained neural NER model.

[0020] After each iteration, the system 100 determines whether the target performance of the neural NER model 315 has been achieved (step 419). If the target performance has not been achieved, the system 100 performs another iteration of retraining the neural NER model 315. Once the system 100 determines that the target performance has been achieved, the training is complete (step 421). In some implementations, the neural NER model 315 can then be further trained using the method of FIG. 4 and a different set of unlabeled data, while in other implementations, the trained neural NER model 315 can be ready for use.

[0021] The training framework in Figure 3 can iteratively discover new entities from an unlabeled corpus by leveraging the benefits of both explicit logic rules and neural representations. Because the framework uses logic rules to obtain "weak" labels and recognize entities, each system prediction can be traced back to the original logic rules, making the prediction results explainable.

[0022] In some implementations, the rule candidate generator 303 is configured to use rule templates (e.g., atomic rules and combination rules) to extract possible rule candidates from the unlabeled data 305. An "atomic rule" is a rule that can be used to describe a single signal aspect of a candidate entity, and a "combination rule" is a rule that can be used to match multiple aspects of an entity. In some implementations, the atomic rules ti is a rule template t i The atomic rule template is generated from the atomic rule. Every atomic rule is associated with an entity label. Examples of atomic rule templates include: (1) SurfaceForm (surface name that matches the entire name of a given entity) (e.g., if x matches "London", then x is LOC), (2) Prefix (matches the prefix of the candidate span) (e.g., if x matches "Lon*", then x is LOC); (3) Suffix (matches the suffix of the candidate span) (e.g., if x matches "*don", then x is LOC); (4) PreNgram (matches the left context of the candidate span) (e.g., "located at x", where x is LOC); (5) PostNgram (matches the right context of the candidate span) (e.g., in “xtown”, x is LOC); (6) POStag (matching the part-of-speech pattern of the candidate span), and (7) PreDependency (parents and siblings of the span on the dependency tree), Includes:

[0023] For example, consider the sentence "The new company is called AdOn GmbH and is located in Hamburg." Using the PreNgram rule "company is called{*}" will result in matches of spans such as [AdOn], [AdOn GmbH], and [AdOn GmbH and], up to the maximum span length. Thus, using only atomic rules will result in many "noisy" spans (i.e., spans that are incorrectly identified as "named entities" by the atomic rules).

[0024] A combination rule is a combination of multiple atomic rules using the logical conjunction "^", logical disjunction "v" or other logical operators. r composed =L(r1,r2,…,r n ) (1) where r1,r2,…,r n are atomic rules, and L is a logical function for connecting atomic rules. Consider again the sentence above, "The new company is called AdOn GmbH and is located in Hamburg." If "PROPN" denotes the part-of-speech tag for the appropriate noun, and the template (PreNgram^POStag) gives the connection rule "(company is called{*},PROPN)", then an exact match with the entity [AdOn GmbH] will be obtained.

[0025] Thus, in some implementations, for every candidate entity, the rule candidate generator 303 extracts all of its rules according to a given rule template. Different rules may be valid for different domains. Thus, the system is potentially configurable to use different types of rules for different target domains. For example, in some biomedical domain datasets, prefix and suffix rules are more efficient rule templates than part-of-speech tags. In some implementations, the framework illustrated in the example of FIG. 3 above allows users to customize their rule templates according to their dataset and domain.

[0026] As described above, the rule labeler 309 is configured to receive a set of unlabeled candidate entities (i.e., spans) and a set of labeling rules 311 and apply the labeling rules to the unlabeled spans to obtain weakly labeled data. In some situations, different rules may produce different labels for the same candidate entity. Therefore, in some implementations, the system 100 is configured to use majority voting to address rule conflicts. For example, if a candidate entity matches a total of three rules, and two rules label the candidate entity as a “location,” while a third rule labels the entity as an “organization,” the system 100 assigns the “location” label to the candidate entity using a majority vote. In some implementations, if the same number of rules apply different labels to the candidate entity (e.g., “tie”), the system 100 is configured to label the candidate entity as an “abstain,” meaning that the candidate entity is not assigned a label for training the neural NER model 315.

[0027] As described above with reference to FIG. 2, in some implementations, the machine learning model 201 (e.g., the neural NER model 315) is configured to produce as output a set of distinct spans and a label assigned to each output span. i and e i are the start and end indices, respectively, of a sentence x=[w1, w2, …, w n ] and span

number

number

number

number

[0028] In some implementations, the neural NER model 315 is configured to use a multi-layer perceptron (MLP) to predict labels for all spans up to a fixed length of l words, i.e. o i = softmax(MLP span (z i )) (7) where o iis a prediction of the span. As discussed above, in some implementations, the negative label NEG is used as an additional label to indicate invalid spans (e.g., spans that are not named entities in the unlabeled data).

[0029] As described above, the candidate rule generator 303 is configured to generate all candidate rules from the unlabeled data using a predetermined rule template. In some implementations of the learning framework of FIG. 3, the system is configured to automatically select a new labeling rule from the candidate rules using the rule selector 317. At each iteration, the rule selector 317 scores all candidate rules and selects the rule with the highest score as the new labeling rule 311. For example, in some implementations, the system first estimates possible labels for all candidate spans using the trained neural NER model 315 (i.e., the set of "predicted data" labels), and then scores each candidate rule by comparing the results of each rule with its weakly estimated label. In some implementations, each candidate rule r i The score for the equation

number

[0030] In some implementations, the system is configured to identify, for each rule template and each entity category, a defined number (N) of the rules with the highest scores as new labeling rules for the next iteration. In some implementations, the system is configured to use N=5 for the first iteration. In some implementations, the system is also configured to prevent low-accuracy rules from being added to the pool of labeling rules by setting a threshold (τ=0.8) for rule accuracy. This method allows for consideration of a variety of patterns while still providing sufficient accuracy for all patterns to be strongly associated with entity categories.

[0031] Thus, the systems and methods described in the above embodiments provide a mechanism for weakly supervised learning of machine learning-based named entity recognition (NER) models, particularly by iteratively scoring a set of automatically generated rule candidates against a trained NER model and using the rule candidates with the highest scores to generate training data labels for subsequent retraining iterations of the NER model. Features and advantages of the invention are set forth in the following claims.

Claims

1. 1. A method for training a machine learning model to perform named entity recognition of unlabeled text data, the machine learning model being configured to receive as input unlabeled text data and produce as output one or more text spans identified as named entities and a category label assigned to each of the one or more text spans identified as named entities, the method comprising: identifying, by an electronic processor, a plurality of entity candidates from an input dataset of unlabeled text, each entity candidate comprising a text span from said input dataset; applying, by the electronic processor, a set of seeding rules to the input data set to assign a label to each entity candidate of the plurality of entity candidates; training, by the electronic processor, the machine learning model using labels assigned to the plurality of candidate entities using the set of seeding rules as a first set of training data; applying, by the electronic processor, the trained machine learning model to the unlabeled text of the input dataset to generate a first set of prediction data including the one or more text spans from the unlabeled text identified as named entities and the category labels assigned to each of the one or more text spans identified as named entities; automatically generating, by the electronic processor, a plurality of candidate rules from the input dataset of unlabeled text, each candidate rule being generated using a set of rule templates; applying, by the electronic processor, the plurality of candidate rules to the unlabeled text of the input data set to generate a set of candidate label data; identifying, by the electronic processor, a subset of rules from the plurality of candidate rules that, when applied to the unlabeled text, assign labels to entity candidates that most closely match assigned labels in the first set of prediction data; retraining, by the electronic processor, the machine learning model using the labels assigned by the identified subset of rules as a second set of training data; A method comprising: applying, by the electronic processor, the retrained machine learning model to the unlabeled data of the input dataset to generate a second set of predicted data; and identifying, by the electronic processor, a second subset of rules from the plurality of candidate rules that, when applied to the unlabeled text, assign labels to entity candidates that most closely match assigned labels in the second set of prediction data; further retraining, by the electronic processor, the machine learning model using labels assigned by the identified second subset of rules as a third set of training data; and The method of claim 1 further comprising: applying, by the electronic processor, the retrained machine learning model to the unlabeled text of the input dataset; identifying, by the electronic processor, a subsequent subset of rules from the plurality of candidate rules that, when applied to the unlabeled text, assign labels to entity candidates that most accurately match labels assigned by the retrained machine learning model; performing additional and iterative retraining of the machine learning model using labels assigned by subsequent subsets of the rules as subsequent sets of training data, by the electronic processor; and iteratively retraining, by the electronic processor, the machine learning model by repeating The method of claim 1 further comprising:

4. Identifying, by the electronic processor, the plurality of entity candidates from the input dataset of unlabeled text includes: a text span for each distinct word in the unlabeled text; and a text span for each sequence of two or more consecutive adjacent words in the unlabeled text; as a candidate entity by the electronic processor.

5. The method of claim 1, wherein applying the set of seeding rules to the input data set by the electronic processor includes applying a small set of predetermined symbolic rules to the input data set by the electronic processor.

6. The method of claim 1, wherein automatically generating, by the electronic processor, a plurality of rule candidates from the input dataset of unlabeled text includes generating, by the electronic processor, for each entity candidate of the plurality of entity candidates, all possible rules defined by each rule template in the set of rule templates.

7. Automatically generating, by the electronic processor, a plurality of candidate rules from the input dataset of unlabeled text includes generating, by the electronic processor, a plurality of atomic rules and a plurality of combination rules for each of a plurality of category labels; each atomic rule of the plurality of atomic rules assigns a category label to each candidate entity that satisfies a single matching criterion defined by the atomic rule; each combination rule of the plurality of combination rules assigns a category label to each entity candidate that satisfies a matching criterion defined by a different combination of two or more atomic rules; The method of claim 1.

8. Identifying, by the electronic processor, a subset of rules from the plurality of candidate rules that assign labels that most closely match assigned labels in the first set of prediction data includes: assigning, by the electronic processor, a score to each rule candidate based on the number of entity candidates assigned the same label by the rule candidate and by the trained machine learning model; including, by the electronic processor, rules having a defined number of highest scores for each category label of a plurality of category labels in the identified subset of rules; The method of claim 1 , comprising:

9. Identifying, by the electronic processor, a subset of rules from the plurality of candidate rules that assign labels that most accurately match assigned labels in the first set of prediction data includes: ｉ is the number of entity candidates assigned the same label by the rule candidate and by the trained machine learning model, and N ｉ is the total number of entity candidates assigned the same label by the rule candidate, and the electronic processor calculates a ratio F ｉ / N ｉ 2. The method of claim 1, comprising assigning a score to each candidate rule based at least in part on:

10. The electronic processor assigning the score to each candidate rule is performed by: F(r ｉ ) is the rule candidate r ｉ is a score assigned by said electronic processor according to the formula [Equation 1] 10. The method of claim 9, further comprising assigning a score to each candidate rule based on:

11. 1. A computer-based system for training a machine learning model to perform named entity recognition of unlabeled text data, the machine learning model being configured to receive as input unlabeled text data and produce as output one or more text spans identified as named entities and a category label assigned to each of the one or more text spans identified as named entities, the system comprising: The system includes an electronic processor, the electronic processor comprising: Identifying a plurality of entity candidates from an input dataset of unlabeled text, each entity candidate comprising a text span from the input dataset; applying a set of seeding rules to the input dataset to assign a label to each entity candidate of the plurality of entity candidates; training the machine learning model using labels assigned to the plurality of entity candidates using the set of seeding rules as a first set of training data; applying the trained machine learning model to the unlabeled text of the input dataset to create a first set of predicted data, the first set including the one or more text spans from the unlabeled text identified as named entities and the category labels assigned to each of the one or more text spans identified as named entities; automatically generating a plurality of candidate rules from the input dataset of unlabeled text, each candidate rule being generated using a set of rule templates; applying the plurality of candidate rules to the unlabeled text of the input dataset; identifying a subset of rules from the plurality of candidate rules that, when applied to the unlabeled text, assign labels to entity candidates that most accurately match assigned labels in the first set of prediction data; retraining the machine learning model using the labels assigned by the identified subset of rules as a second set of training data. The system is configured as follows:

12. The electronic processor applying the retrained machine learning model to the unlabeled data of the input dataset to generate a second set of predicted data; identifying a second subset of rules from the plurality of candidate rules that, when applied to the unlabeled text, assign labels to entity candidates that most accurately match assigned labels in the second set of prediction data; Further retraining the machine learning model using the labels assigned by the identified second subset of rules as a third set of training data. The system of claim 11 further configured to:

13. The electronic processor applying the retrained machine learning model to the unlabeled text of the input dataset; identifying a subsequent subset of rules from the plurality of candidate rules that, when applied to the unlabeled text, assign labels to entity candidates that most accurately match labels assigned by the retrained machine learning model; and performing additional and iterative retraining of the machine learning model using the labels assigned by subsequent subsets of the rules as subsequent sets of training data; 12. The system of claim 11, further configured to iteratively retrain the machine learning model by repeating:

14. The electronic processor a text span for each distinct word in the unlabeled text; and a text span for each sequence of two or more consecutive adjacent words in the unlabeled text; as entity candidates, thereby identifying the plurality of entity candidates from the input dataset of unlabeled text. The system of claim 11 further configured to:

15. 12. The system of claim 11, wherein the electronic processor is further configured to apply the set of seeding rules to the input data set by applying a subset of predetermined symbolic rules to the input data set.

16. 12. The system of claim 11, wherein the electronic processor is further configured to automatically generate a plurality of rule candidates from the input dataset of unlabeled text by generating, for each entity candidate of the plurality of entity candidates, all possible rules defined by each rule template in a set of rule templates.

17. the electronic processor is further configured to automatically generate a plurality of candidate rules from the input dataset of unlabeled text by generating a plurality of atomic rules and a plurality of combination rules for each of a plurality of category labels; each atomic rule of the plurality of atomic rules assigns a category label to each candidate entity that satisfies a single matching criterion defined by the atomic rule; each combination rule of the plurality of combination rules assigns a category label to each entity candidate that satisfies a matching criterion defined by a different combination of two or more atomic rules; The system of claim 11.

18. The electronic processor assigning a score to each rule candidate based on the number of entity candidates assigned the same label by the rule candidate and by the trained machine learning model; and including within the identified subset of rules a defined number of rules having the highest scores for each of the plurality of category labels. identifying a subset of rules from the plurality of candidate rules that assign labels that most closely match assigned labels in the first set of prediction data by The system of claim 11 further configured to:

19. The electronic processor is ｉ is the number of entity candidates assigned the same label by the rule candidate and by the trained machine learning model, and N ｉ is the total number of entity candidates to which the same label is assigned by the rule candidate, and the ratio F ｉ / N ｉ 12. The system of claim 11, further configured to identify a subset of rules from the plurality of candidate rules that assign labels that most accurately match assigned labels in the first set of prediction data by assigning a score to each candidate rule based at least in part on:

20. The electronic processor calculates F(r ｉ ) is the rule candidate r ｉ is the score assigned to [Equation 2] and further configured to assign a score to each candidate rule by assigning the score to each candidate rule based on 20. The system of claim 19.

Images