A method and apparatus for training a named entity recognition model

Abstract

The application provides a training method and device of a named entity recognition model, which can train the named entity recognition model based on a small amount of training data samples. The method comprises the following steps: obtaining a seed template, the seed template being an abstract semantic representation of a corpus used for training the named entity recognition model, the seed template comprising one or more first data filling functions, the first data filling function being used for obtaining a named entity corresponding to a named entity label according to the named entity label to form a first corpus, the seed template being used for determining the first corpus according to the named entity and determining annotation information of the first corpus, the annotation information comprising a position of the named entity and a named entity label corresponding to the named entity, the named entity label being used as an expected output of the named entity recognition model to train the model; determining a training data set according to the first corpus and the annotation information; and training the named entity recognition model based on the training data set to obtain a trained named entity recognition model.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a training method and apparatus for a named entity recognition model. Background Technology

[0002] Named entity recognition, also known as proper name recognition, refers to the identification of entities with specific meanings in text. Entities mainly include names of people, places, organizations, and proper nouns. Named entity recognition models are important foundational tools in application areas such as information extraction, question answering systems, syntactic analysis, machine translation, and metadata annotation for the Semantic Web, playing a crucial role in the practical application of natural language processing technology.

[0003] Unlike the standardized datasets used in academia, industrial datasets are often more complex and have higher business implications. This makes data annotation in industrial scenarios more difficult and prone to errors, and these problems are even more pronounced in the industrial applications of today's named entity recognition models.

[0004] Therefore, how to train a named entity recognition model based on a small number of training data samples has become an urgent problem to be solved. Summary of the Invention

[0005] This application provides a training method and apparatus for a named entity recognition model, which can train the named entity recognition model based on a small number of training data samples, thereby enhancing the training data and improving the model's generalization ability.

[0006] Firstly, this application provides a method for training a named entity recognition model, including:

[0007] A seed template is obtained, which is an abstract semantic representation of the corpus used to train the named entity recognition model. The seed template includes one or more first data filling functions, which are used to obtain the named entities corresponding to the named entity labels that make up the first corpus based on the named entity labels. The seed template is used to determine the first corpus based on the named entities and to determine the annotation information of the first corpus. The annotation information includes the position of the named entity and the named entity label corresponding to the named entity. The named entity label is used as the expected output of the named entity recognition model to train the model.

[0008] The training dataset is determined based on the first corpus and the annotation information;

[0009] The named entity recognition model is trained based on this training dataset to obtain a well-trained named entity recognition model.

[0010] In one example, the method also includes:

[0011] Obtain the thesaurus, which stores different named entity category names and the named entities corresponding to each named entity category name.

[0012] In one example, the named entity category names stored in the lexicon include the named entity label. The first data filling function is used to obtain the named entity corresponding to the named entity label that makes up the corpus, including:

[0013] The first data filling function is used to associate the named entity tag with the thesaurus and to obtain the named entity in the thesaurus that corresponds to the named entity tag.

[0014] In one example, the seed template includes one or more second data filling functions, each comprising multiple sub-functions. These second data filling functions are used to determine and execute a sub-function from among these sub-functions.

[0015] The plurality of sub-functions includes a first sub-function, which is used to obtain named entities that make up the first corpus, excluding the named entities included in the vocabulary. The seed template is also used to determine the first corpus based on the named entities obtained by the first sub-function, and / or,

[0016] The plurality of sub-functions includes a second sub-function, which is used to obtain the word segments that make up the first corpus, excluding named entities. The seed template is also used to determine the first corpus based on the word segments.

[0017] In one example, the method also includes:

[0018] Obtain seed data, which is the original corpus after being manually annotated with the named entity label, and the seed data includes the annotation information;

[0019] The second corpus was determined based on the annotation information of the seed data and the vocabulary.

[0020] The training dataset is determined based on the second corpus and the annotation information.

[0021] In one example, the method also includes:

[0022] Obtain historical datasets, which include the datasets used by the trained named entity recognition model and the metadata of the datasets used. The metadata includes one or more named entity labels in the dataset.

[0023] Based on the metadata, obtain the same labeled dataset in the historical dataset. The named entity labels of the same labeled dataset are the same as those of the first corpus. The same labeled dataset includes the annotation information.

[0024] The third corpus was determined based on the annotation information of the same dataset and the vocabulary.

[0025] The training dataset is determined based on the third corpus and the annotation information.

[0026] In one example, the method also includes:

[0027] Obtain the fourth corpus including this annotation information;

[0028] Based on the annotation information of the fourth corpus, determine the new first data filling function that matches the named entity corresponding to the annotation information;

[0029] A new seed template is determined based on the new first data filling function, and this new seed template is used to determine the training dataset for subsequent named entity recognition model training.

[0030] In one example, determining the new seed template based on the new first data fill function includes:

[0031] Identify word segments other than named entities in this fourth corpus;

[0032] Based on the word segmentation of the fourth corpus (excluding named entities), a new second data filling function is determined using regular expression matching.

[0033] The new seed template is determined based on the new first data filling function and the new second data filling function.

[0034] In one example, the method also includes:

[0035] Validate the trained named entity recognition model.

[0036] Update the seed template based on the results of the model validation;

[0037] The new training dataset is determined based on the updated seed template;

[0038] The named entity recognition model is trained based on this new training dataset to obtain a new, well-trained named entity recognition model.

[0039] Secondly, this application provides a training apparatus for a named entity recognition model, comprising:

[0040] A seed template acquisition module acquires a seed template, which is an abstract semantic representation of the corpus used to train a named entity recognition model. The seed template includes one or more first data filling functions, which are used to acquire named entities corresponding to the named entity tags that make up the first corpus based on the named entity tags. The seed template is used to determine the first corpus based on the named entities and to determine the annotation information of the first corpus. The annotation information includes the position of the named entity and the named entity tag corresponding to the named entity. The named entity tag is used as the expected output of the named entity recognition model to train the model.

[0041] The training dataset determination module is used to determine the training dataset based on the first corpus and the annotation information;

[0042] The model training module is used to train the named entity recognition model based on the training dataset to obtain the trained named entity recognition model.

[0043] In one example, the device also includes a thesaurus acquisition module for acquiring a thesaurus that stores different named entity category names and the named entities corresponding to each named entity category name.

[0044] In one example, the device also includes a seed data acquisition device for acquiring seed data, which is the original corpus after being manually annotated with the named entity tags, and the seed data includes the annotation information;

[0045] The corpus determination module is used to determine the second corpus based on the annotation information of the seed data and the vocabulary.

[0046] The training dataset determination module is also used to determine the training dataset based on the second corpus and the annotation information.

[0047] In one example, the device also includes a historical dataset acquisition module for acquiring a historical dataset, which includes the dataset used by the pre-trained named entity recognition model and the metadata of the dataset, which includes one or more named entity labels in the dataset.

[0048] The same-label dataset acquisition module is used to obtain the same-label dataset from the historical dataset based on the metadata. The named entity labels of the same-label dataset are the same as those of the first corpus. The same-label dataset includes annotation information.

[0049] The corpus determination module is also used to determine a third corpus based on the annotation information of the same-label dataset and the vocabulary;

[0050] The training dataset determination module is also used to determine the training dataset based on the third corpus and the annotation information.

[0051] In one example, the device further includes: a corpus determination module is also used to acquire a fourth corpus including the annotation information;

[0052] The data filling function determination module is used to determine a new first data filling function that matches the named entity corresponding to the annotation information based on the annotation information of the fourth corpus.

[0053] The seed template acquisition module is also used to determine a new seed template based on the new first data filling function. This new seed template is used to determine the training dataset for subsequent named entity recognition model training.

[0054] The model validation module is used to validate the trained named entity recognition model.

[0055] The seed template acquisition module is also used to update the seed template based on the results of the model validation;

[0056] The training dataset determination module is also used to determine a new training dataset based on the updated seed template;

[0057] The named entity recognition model is trained based on this new training dataset to obtain a new, well-trained named entity recognition model.

[0058] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the methods provided in any of the above embodiments.

[0059] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the methods provided in any of the above embodiments.

[0060] As described in the above embodiments, this application provides a seed template for generating text data that constitutes a corpus. This seed template is an abstract semantic representation of the corpus. Its data filling function can obtain the named entities constituting the first corpus based on the named entity labels used in training the named entity recognition model. The seed template determines new corpus based on the obtained named entities and can grasp the annotation information of the corpus. Therefore, it can determine the training dataset based on the new corpus and its annotation information, so as to facilitate subsequent training of the named entity recognition model based on this training dataset. This method can not only use the seed template to generate various corpora with annotation information to expand the training dataset, improving the diversity of the training dataset and thus improving the generalization ability of the final model, but also save the time and manpower wasted on manually annotating a large amount of data. Furthermore, in industrial business scenarios where confidential data cannot be obtained, users can construct a seed template according to the format of the seed template in this application based on their confidential data and send it to the model training backend, where the model training backend runs the seed template to generate the corpus required for the training dataset. This further improves the effect of training data augmentation and solves the problem of starting named entity recognition model training when the data ontology is not visible. Attached Figure Description

[0061] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a schematic diagram of a training method for a named entity recognition model provided in some embodiments of this application;

[0063] Figure 2 This is a schematic diagram of the training method of another example of a named entity recognition model provided in some embodiments of this application;

[0064] Figure 3 This is a schematic diagram of a system framework for training a named entity recognition model provided in some embodiments of this application. Detailed Implementation

[0065] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0066] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0067] To facilitate understanding of the proposed solution, some technical concepts will be briefly introduced below.

[0068] Corpus: Typically, it is a collection of text resources of a certain quantity and scale. The size of a corpus can vary greatly, ranging from tens of millions or even hundreds of millions of sentences or more, to as few as a few hundred sentences.

[0069] Named entity: refers to the name or symbol of a specific type of thing, usually a word or noun phrase. The term named entity is often extended to include things that are not entities in themselves, including dates, times and other kinds of time expressions, and even numerical expressions like prices.

[0070] Named entity labels: The category names of named entities used when labeling entities in a corpus. Named entity labels are generally determined based on the specific task of the named entity recognition model.

[0071] In the industrial applications of machine learning and deep learning in natural language processing, the limited number of samples and the difficulty of sample annotation have always been major challenges. The datasets used for training models in the industrial application of named entity recognition (NER) are often more complex and have higher business implications, differing significantly from the standardized datasets used in academia. Furthermore, in some business scenarios, the data itself is not publicly available (e.g., in public security, banking, etc.). This makes named entity annotation more difficult, more prone to errors, and manually annotating large amounts of specific business data is obviously both time-consuming and labor-intensive. Therefore, how to enhance the training of named entity recognition models with small sample data, or even when sample data is unavailable, has become an urgent problem to be solved.

[0072] To address the aforementioned issues, this application proposes a training method for a named entity recognition model, see [link to relevant documentation]. Figure 1 The method includes:

[0073] S110, Obtain a seed template, which is an abstract semantic representation of the corpus used to train the named entity recognition model.

[0074] Since the named entities contained in the corpus can be abstracted into named entity tags (which are also abstract semantic representations of the named entities), a corpus formed by replacing the named entity portion of the corpus with its corresponding named entity tags is a corpus in an abstract sense. Similarly, in addition to named entities, the corpus also includes word segments of various parts of speech, and the word segment portion of the corpus can also be replaced with the abstract (or higher-level) semantic representation of the word segment, such as its corresponding part of speech.

[0075] For example, given the corpus "Xiaoming lives in Anwai Beiyuan," the abstract semantic representation of "Xiaoming" is "person's name," the abstract semantic representation of "lives in" is "preposition," and the abstract semantic representation of "Anwai Beiyuan" (i.e., named entity label) is "landmark." Then, the abstract semantic representation of the corpus derived from this corpus (also called abstract corpus or superordinate corpus) is [person's name][preposition][landmark].

[0076] Based on the abstract semantic representation in the corpus, this application provides a data imputation function, which can generate new text data based on the abstract semantic representation in the corpus. For example, [landmark] is an abstract semantic representation, but the data imputation function f = [landmark] is determined based on this abstract semantic representation. This data imputation function f can generate named entities of other landmark classes of [landmark].

[0077] A single data filling function can only generate its corresponding text data. This application provides a seed template, which can organize the generated text data into a new corpus based on the positional relationship between the various data filling functions included in the seed template. For example, the initial corpus is "Xiaoming lives in Anwai Beiyuan". Based on this initial corpus, the provided seed template m = f1f2f3, where f1 = [person's name], f1 is used to generate various person's names, f2 = [preposition], f2 is used to generate various prepositions, and f3 = [landmark], is used to generate various landmark named entities. The new corpus generated by this seed template is "Xiaohong stays at Xinyuan International".

[0078] Without considering the functions of each data filling function, the seed template, in terms of form, can also be called "[person's name][preposition][landmark]" as an abstract semantic representation of a corpus. That is, the seed template is an abstract semantic representation of a corpus in form, and its function is to generate new corpus.

[0079] Based on the above, this application provides an embodiment in which the seed template includes one or more first data filling functions. The first data filling functions are used to obtain the named entities corresponding to the named entity tags that make up the first corpus according to the named entity tags. The seed template is used to determine the first corpus according to the named entities and to determine the annotation information of the first corpus. The annotation information includes the position of the named entity and the named entity tag corresponding to the named entity. The named entity tag is used as the expected output of the named entity recognition model to train the model.

[0080] Because the data imputation functions include entity category labels (such as "landmark" and "district"), and the seed template, based on the positional relationships of each data imputation function, can naturally obtain entity category labels and corresponding positional information from the new corpus, the corpus generated based on this seed template can be directly used as a training dataset, obtaining a large amount of training data without extensive manual annotation. On the other hand, for industrial application scenarios where named entity data needs to be kept confidential, when the actual data in the industrial business cannot be obtained, users can provide the entity category labels, various word segmentation parts of speech, and the positional relationships of each word segmentation required to construct the seed template based on their confidential data. The seed template can be generated locally from the application or sent to the model training backend, which will then generate the seed template and run it to generate the corpus required for the training dataset.

[0081] For example, the data imputation function also includes an occurrence probability, which represents the probability that the data imputation function will take effect, i.e., how likely it is that the data imputation function will work to obtain text data. If the data imputation function does not take effect, the seed template is replaced with an empty string at its position in the corpus.

[0082] In one example, method S110 also includes: obtaining a thesaurus, which stores different named entity category names and the named entities corresponding to each named entity category name.

[0083] For example, the thesaurus is stored in the form of a database.

[0084] The data source for the lexicon can be manually collected named entities corresponding to a specific named entity category, or manually supplemented named entity categories not present in the original lexicon and their corresponding named entities. Alternatively, the data source for the lexicon can also be named entity categories and their corresponding named entities accumulated after training the model using the scheme described in this application.

[0085] Therefore, this thesaurus is updatable and expandable. The contents of the thesaurus are shown in Table 1 below:

[0086] Table 1

[0087] Named entity category name Named Entities landmark - neighborhood Area A, Area B, JiaX Residential Area, Anwai Beiyuan landmark - building XX Building district XX Road, XX Street fruit Apples, bananas … …

[0088] For example, the named entity category names stored in the dictionary include the named entity labels used in training the named entity recognition model. The first data filling function can be used to associate the named entity labels with the dictionary and obtain the named entities in the dictionary corresponding to the named entity labels. This data filling function can be called the dictionary function.

[0089] For example, as shown in Table 1, the named entity corresponding to "district" in the dictionary includes various types of streets. The first data filling function is f4 = [district#0.5#]. f4 associates the named entity tag "district" with named entities such as "XX Road" and "XX Street" in the dictionary. The probability of this data filling function f4 taking effect is 0.5. In summary, the first data filling function is f = [named entity tag #probability of occurrence#], where f is used to associate the named entities corresponding to this named entity tag in the dictionary.

[0090] Furthermore, as shown in Table 1 under "landmark-cell", the named entity category name may also include a named entity tag (landmark) and the subclass name (cell) corresponding to that named entity tag. In this case, the dictionary function can also associate the named entity tag, the subclass name corresponding to that named entity tag, and the dictionary, and retrieve the named entity in the dictionary that corresponds to that subclass name. Specifically, if the dictionary function has multiple subclasses corresponding to the named entity tag, it is also used to identify one subclass from among the multiple subclasses, associate the named entity tag, the subclass name, and the dictionary, and retrieve the named entity in the dictionary that corresponds to that subclass name.

[0091] For example, the dictionary function f5 = [landmark{community|park}#0.5#] means that the entity label at the current character position is "landmark", the subclass name used to associate with the dictionary is "community" or "park", and the last two # symbols are a value between 0 and 1, indicating the probability that this function will take effect (if it does not take effect, it will be replaced with an empty string). Therefore, this type of dictionary function can be represented as f = [category{subclass1|subclass2}#probability of occurrence#].

[0092] Using the above method, this application embodiment sets up a dictionary function for associating named entity tags with a dictionary. The dictionary function can retrieve named entities corresponding to specified named entity tags in the dictionary, thus facilitating the rapid generation of various named entities. Furthermore, since the named entities obtained in this way correspond to the named entity tags in the dictionary, the obtained named entities can carry their corresponding named entity tags, i.e., their corresponding annotation information, thereby forming a large dataset for training named entity recognition models, saving the manpower and time wasted on manually annotating datasets.

[0093] For example, in addition to named entities, the constituent elements of the corpus also include word segments of various parts of speech, such as verbs, prepositions, etc. Some word segments of different parts of speech do not belong to the concept of named entities. The corpus may also include named entities that are not in the lexicon.

[0094] For example, consider the corpus "I live in Phase II of Anwai Beiyuan, Chaoyang District." The named entities within this corpus are "Chaoyang District" and "Anwai Beiyuan," corresponding to the named entity tags "district" and "landmark," respectively. As shown in Table 1, "Anwai Beiyuan" corresponds to the subclass "community" of "landmark" in the lexicon. "I" is a personal pronoun, "live in" is a preposition, and "Phase II" is not stored in the lexicon. This demonstrates that, in addition to named entities, word segmentation of various parts of speech further enhances the diversity of the corpus.

[0095] To increase the diversity of new corpora generated by the seed template, this application also provides an embodiment. In this embodiment, the seed template includes one or more second data filling functions, each comprising multiple sub-functions. The second data filling function is used to determine and execute one of the multiple sub-functions. The multiple sub-functions include a first sub-function, which is used to obtain named entities other than those included in the lexicon that constitute the first corpus. The seed template is further used to determine the first corpus based on the named entities obtained by the first sub-function. And / or, the multiple sub-functions include a second sub-function, which is used to obtain word segments other than named entities that constitute the first corpus. The seed template is further used to determine the first corpus based on the word segments. This application refers to the second data filling function as a diversity function.

[0096] For example, the diversity function g = [c(function1|function2)#probability#], where c indicates that the function is a diversity function, and function1 and function2 are used to generate named entities not included in the lexicon or word segments other than named entities that make up the corpus. For example, g1 = [c(locative words|regional descriptions)#0.5#], where locative words are sub-functions used to generate words for various locations, and regional descriptions are sub-functions used to generate words for various regions, such as "Phase II," "Phase III," "Building 1," etc. g1 is used to determine one sub-function from these two sub-functions and execute that sub-function. The diversity function g1 has a 0.5 probability of being effective (if it is ineffective, it will be replaced with an empty string). In actual implementation, each diversity function has a function logic, which can be a piece of code that can randomly generate text of a certain part of speech or enumerate some text.

[0097] Using the above method, the embodiments of this application set up a diversity function that can generate named entities not included in the vocabulary of the corpus, or generate word segmentation other than named entities that make up the corpus. This function can generate text data that make up the corpus without relying on named entity tags, which further improves the diversity of the corpus generated by the seed template, strengthens the function of the seed template, and makes the content of the obtained training dataset richer.

[0098] In one example, this application also provides a method for determining a seed template, including:

[0099] A fourth corpus containing the annotation information is obtained. Then, a new first data filling function matching the named entity corresponding to the annotation information is determined based on the annotation information of the fourth corpus. A new seed template is then determined based on the new first data filling function. The new seed template is used to determine the training dataset for subsequent training of the named entity recognition model.

[0100] For example, the correspondence between named entities and the first data filling function can be pre-stored, as shown in Table 2 below:

[0101] Table 2

[0102] First data filling function name Named Entities district Area A, Area B, JiaX Residential Area, Anwai Beiyuan landmark - park XX Road, XX Street ... …

[0103] Based on the annotation information of the fourth corpus, the named entities included are determined. Then, based on the correspondence in Table 2, a new first data filling function name is determined. Finally, a new seed template is determined according to the position of the corresponding text in the corpus based on this new first data filling function. As shown in Table 2, the first data filling function name is either the name of the named entity label or the name of the named entity label plus the name of the subclass to which the named entity belongs.

[0104] In the above embodiments, since the annotation information includes named entity tags, a pre-constructed lexicon function or diversity function corresponding to the named entity tag can be matched based on the named entity tag. A seed template can be generated based on the lexicon function or diversity function and the position of the corresponding named entity tag in the corpus. Since the fourth corpus with annotation information can be the corpus in the training dataset used for training the named entity recognition model, and this training dataset, after being expanded by means of seed templates, includes many new corpora not found in the lexicon or historical datasets, the seed templates generated based on these new corpora can further expand the original seed templates. This effectively utilizes the corpus in the training dataset used for this training, enriches the seed templates used to determine the training dataset before the next model training, and improves the effect of training data augmentation.

[0105] For example, considering that the fourth corpus also includes word segments other than named entities, this application also provides a way to determine the seed template: first, determine the word segments other than named entities in the fourth corpus, then determine a new second data filling function based on the word segments other than named entities in the fourth corpus using a regular expression matching method, and then determine a new seed template based on the new first data filling function and the new second data filling function.

[0106] For example, the correspondence between the set regular expressions and data filling functions is shown in Table 3 below:

[0107] Table 3

[0108] Second data filling function name Regular expressions ID number (c_ID number prefix)

(c_ID number)

(c_ID number)

[0109] Based on the regular expressions in Table 2, the regular expression matching method is used to determine the text that matches the fourth corpus. If the fourth corpus includes ID card numbers, the regular expressions that match will include "(c_ID card number prefix)

(c_ID card number)

(c_ID card number)

ID card number

ID card number

[0110] For example, to further enhance the diversity of the constituent corpus, the seed template provided in this application also includes characters. For example, the seed template is [landmark] for [person's name] [preposition] [district].

[0111] S120, Determine the training dataset based on the first corpus and the annotation information.

[0112] To further enrich the content of the training dataset, this application also provides a method for determining the training dataset, including:

[0113] Seed data is obtained, which is the original corpus after being manually annotated with named entity tags. This seed data includes the annotation information. Then, based on the annotation information of the seed data and the vocabulary, a second corpus is determined. Finally, based on the second corpus and the annotation information, a training dataset is determined. Here, the original corpus refers to the corpus that has not been manually annotated.

[0114] For example, firstly, the position of the named entity label in the seed data is determined based on the annotation information of the seed data. Then, the corresponding named entity in the vocabulary is determined based on the named entity label. The original named entity at the position of the named entity label is replaced with the named entity, thereby generating a new corpus (i.e., the second corpus). The new corpus and its annotation information are then added to the training dataset.

[0115] Furthermore, before determining the training dataset based on the second corpus and the annotation information, it is first determined whether the second corpus is the same as the first corpus. If they are the same, the second corpus is re-determined until the second corpus is different from the first corpus.

[0116] For example, the original corpus is "We live in Phase III of Honglian Community, Dongcheng District". The manually labeled named entity tags are: district-Dongcheng District, landmark-Honglian Community. Therefore, the seed data is "We live in [district][landmark]". Based on this seed data, the named entity corresponding to district can be determined in the lexicon as "Xicheng District", and the named entity corresponding to landmark as "Qinhong Community". Then, according to the positions of the named entity tags in the seed data, the new corpus generated is "We live in Qinhong Community, Xicheng District".

[0117] This application embodiment generates a new corpus based on seed data and a lexicon. Since the annotation information in the new corpus is the same as that in the seed data, it can be used as a training dataset, further enriching the diversity of the training dataset.

[0118] In one example, this application also provides a method for determining the training dataset, including:

[0119] First, the historical dataset is acquired. This historical dataset includes the datasets used by the pre-trained named entity recognition model and the metadata of those datasets. The metadata includes one or more named entity labels from the datasets in question. Next, a homologous dataset is obtained from the historical dataset based on the metadata. The named entity labels in this homologous dataset are identical to those in the first corpus. Specifically, based on one or more named entity labels in the metadata, the homologous labels that match the named entity labels in the first corpus are identified. These homologous labels, along with their corresponding named entities, form the homologous dataset, which includes the annotation information. Then, based on the annotation information in this homologous dataset, the named entity labels and their location information are determined. The corresponding named entities in the vocabulary are then identified based on these named entity labels. The third corpus is determined based on the named entities and their location information. Finally, the training dataset is determined based on the third corpus and the annotation information.

[0120] The metadata of a dataset includes one or more named entity labels from its dataset, as well as a dataset identifier, dataset name, and data source. Data sources can include manually labeled data, manually labeled derived data, historical data, historical data derived data, and template derived data. Manually labeled data indicates training data with manually labeled named entities. Manually labeled derived data indicates data generated using manually labeled data as seed data. Historical data indicates data obtained from historical datasets. Historical data derived data indicates data generated using data from the same labeled dataset as seed data. Template derived data indicates data generated based on a seed template. The dataset identifier and dataset name in the metadata identify the dataset to which it belongs, while the data source in the metadata identifies data from different sources, allowing for targeted optimization of different source data portions in historical datasets based on their origins.

[0121] This application embodiment expands the training dataset by using the dataset already trained using the named entity recognition model according to this method as the historical dataset, and obtaining a dataset with the same labels of the named entities used in this training from the historical dataset as part of the training dataset used in this training, thereby further enriching the diversity of the training dataset and further improving the effect of dataset augmentation.

[0122] S130, Use the training dataset to train the named entity recognition model to obtain the trained named entity recognition model.

[0123] As described in the above embodiments, this application provides a seed template for generating text data that constitutes a corpus. This seed template is an abstract semantic representation of the corpus. Its data filling function can obtain the named entities constituting the first corpus based on the named entity labels used in training the named entity recognition model. The seed template determines new corpus based on the obtained named entities and can grasp the annotation information of the corpus. Therefore, it can determine the training dataset based on the new corpus and its annotation information, and train the named entity recognition model based on this training dataset. This method not only utilizes the seed template to generate various corpora with annotation information to expand the training dataset, improving the diversity of the training dataset and thus enhancing the generalization ability of the final model, but also saves the time and manpower wasted on manually annotating large amounts of data. Furthermore, in industrial business scenarios where confidential data cannot be obtained, users can construct a seed template according to the format of the seed template in this application based on their confidential data, and send it to the model training backend. The model training backend then runs the seed template to generate the corpus required for the training dataset. This further improves the effect of training data augmentation and solves the problem of starting named entity recognition model training when the data ontology is not visible.

[0124] In one example, see Figure 2 Training methods for named entity recognition models also include:

[0125] S140, Perform model validation on the trained named entity recognition model.

[0126] Specifically, first, a validation dataset is obtained. The corpus from the validation dataset is then input into the named entity recognition model for inference, yielding the inference results. Next, the inference results are compared with the named entity labels in the validation dataset to calculate evaluation metrics such as precision, recall, and F1 score. These metrics determine whether further model optimization is necessary (e.g., an accuracy exceeding 95% is sufficient for commercial applications and further optimization is not required).

[0127] S150, Update the seed template based on the results of the model validation.

[0128] Specifically, if iterative optimization of the model is required, the corpus in the validation dataset where the model inference results and named entity labels are inconsistent is selected, and the seed template is updated based on this corpus. The model validation results include the corpus in the validation dataset where the model inference results and named entity labels are inconsistent.

[0129] S160, determine the new training dataset.

[0130] Specifically, the updated seed template is used to regenerate the corpus and the corresponding annotation information, and the training dataset is redefined based on the new corpus and the new annotation information.

[0131] S170, retrain the named entity recognition model.

[0132] Specifically, the named entity recognition model is trained based on the new training dataset to obtain a new trained named entity recognition model.

[0133] The following example illustrates the process of optimizing a named entity recognition model:

[0134] The original seed template is: [c(person reference)] lives in [landmark{circle area|sightseeing spot}#0.5#][landmark{neighborhood}][c(separator)][c(location reference)][c(existential quantifier)][landmark{shopping mall|supermarket|building}]

[0135] The corpus for the validation dataset includes: I live in Phase III of Lin'ao Jiayuan, near the entrance of Olympic Forest Park, where there is a Hema Fresh supermarket.

[0136] The named entity tags of the above corpus in the validation dataset are: landmark-Aosen Park, Lin'ao Garden, and Hema Fresh. When the above validation dataset corpus is input into the model, the resulting inference is: landmark-Aosen Park.

[0137] As can be seen, the model's inference results are different from the named entity labels on the validation set. Therefore, the new seed template determined based on this corpus is: [c(personal pronoun)] lives in [landmark{circle area|attraction}] in [landmark{community}], and there is a [landmark{shopping mall}] at the entrance.

[0138] Using the above method, the corpus in the validation dataset is input into the named entity recognition model. The obtained model inference results are compared with the named entity labels in the validation dataset to obtain the model validation results. If iterative optimization of the model is required, the seed template is updated based on the corpus in the model validation results, and new corpus is generated based on the updated seed template, thereby updating the training dataset and improving the effect of training dataset augmentation. The model is then trained again based on the updated training dataset to obtain a more accurate model.

[0139] In conjunction with the above embodiments, such as Figure 3As shown, this application provides a schematic diagram of a system framework for training a named entity recognition model. The knowledge accumulation unit includes a section for storing a lexicon, a section for storing seed templates, and a section for storing historical datasets. The data input to the input unit includes seed templates, seed data, and a homologous dataset. The training dataset generation unit can generate training data in three ways: combining seed templates and a lexicon, combining seed data and a lexicon, or combining a homologous dataset and a lexicon. The named entity recognition model training unit trains the model using the training data in the training dataset, obtaining a trained named entity recognition model after multiple iterations. The model validation unit uses a test dataset to obtain the model's inference results. These inference results are used by the model evaluation unit to assess the model's performance. If the model meets commercial requirements, it is output, and the process ends. If the model does not meet commercial requirements, the template optimization unit updates the seed template. Figure 3 As can be seen, the training data in each generated training dataset can be added to the historical dataset, and seed templates can be extracted from the training dataset to enrich the seed template data stored in the knowledge accumulation unit. This allows for the continuous expansion of the historical dataset, enriching the knowledge of the entire system and facilitating reuse for other similar needs in the future.

[0140] There are several methods for determining the test dataset in the model review unit, such as: 1. Manually annotated corpora can be used directly as the test dataset; 2. The named entity recognition model can first annotate the corpora and then submit them to human reviewers, with the annotated corpora after human review being used as the test dataset; 3. When multiple training processes are triggered in the model training process, i.e., there are multiple models using the same named entity labels, these multiple models can also be used to annotate the corpora. In this case, human reviewers can focus on the corpora whose inference results are inconsistent among the multiple models and correct the annotation results; 4. The corpora can also be used to retrieve named entity labels that match the corpora from the dictionary, and then manually reviewed and modified before being used as the test set.

[0141] In conjunction with the above-described training method embodiments for named entity recognition models, this application also provides a training apparatus for named entity recognition models, comprising:

[0142] The seed template acquisition module acquires a seed template, which is an abstract semantic representation of the corpus used to train the named entity recognition model. The seed template includes one or more first data filling functions, which are used to acquire the named entities corresponding to the named entity labels that make up the first corpus based on the named entity labels. The seed template is used to determine the first corpus based on the named entities and to determine the annotation information of the first corpus. The annotation information includes the position of the named entity and the named entity label corresponding to the named entity. The named entity label is used as the expected output of the named entity recognition model to train the model.

[0143] The training dataset determination module is used to determine the training dataset based on the first corpus and the annotation information.

[0144] The model training module is used to train the named entity recognition model based on the training dataset to obtain the trained named entity recognition model.

[0145] In one example, the device also includes a thesaurus acquisition module for acquiring a thesaurus that stores different named entity category names and the named entities corresponding to each named entity category name.

[0146] In one example, the device also includes a seed data acquisition device for acquiring seed data, which is the original corpus after being manually annotated with the named entity tags, and the seed data includes the annotation information;

[0147] The corpus determination module is used to determine the second corpus based on the annotation information of the seed data and the vocabulary.

[0148] The training dataset determination module is also used to determine the training dataset based on the second corpus and the annotation information.

[0149] In one example, the device also includes a historical dataset acquisition module for acquiring a historical dataset, which includes the dataset used by the pre-trained named entity recognition model and the metadata of the dataset, which includes one or more named entity labels in the dataset.

[0150] The same-label dataset acquisition module is used to obtain the same-label dataset from the historical dataset based on the metadata. The named entity labels of the same-label dataset are the same as those of the first corpus. The same-label dataset includes annotation information.

[0151] The corpus determination module is also used to determine a third corpus based on the annotation information of the same-label dataset and the vocabulary;

[0152] The training dataset determination module is also used to determine the training dataset based on the third corpus and the annotation information.

[0153] In one example, the device further includes: a corpus determination module is also used to acquire a fourth corpus including the annotation information;

[0154] The data filling function determination module is used to determine a new first data filling function that matches the named entity corresponding to the annotation information based on the annotation information of the fourth corpus.

[0155] The seed template acquisition module is also used to determine a new seed template based on the new first data filling function. This new seed template is used to determine the training dataset for subsequent named entity recognition model training.

[0156] The model validation module is used to validate the trained named entity recognition model.

[0157] The seed template acquisition module is also used to update the seed template based on the results of the model validation;

[0158] The training dataset determination module is also used to determine a new training dataset based on the updated seed template;

[0159] The named entity recognition model is trained based on this new training dataset to obtain a new, well-trained named entity recognition model.

[0160] Other implementation methods and corresponding effects are described in the above-described training method embodiments for the named entity recognition model, and will not be repeated here.

[0161] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0162] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0163] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0164] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0165] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0166] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for training a named entity recognition model, the method comprising: The method comprises the following steps: obtaining a vocabulary, the vocabulary being used to store different named entity category names and corresponding named entities of each named entity category name; obtaining a seed template, the seed template being an abstract semantic representation of a corpus used to train a named entity recognition model, the seed template comprising one or more first data filling functions, the first data filling functions being used to obtain, according to a named entity tag, a named entity corresponding to the named entity tag to form a first corpus, the seed template being used to determine the first corpus according to the named entity and determine annotation information of the first corpus, the annotation information comprising a position of the named entity and a named entity tag corresponding to the named entity, the named entity tag being used as an expected output of the named entity recognition model to train the model; the seed template further comprises one or more second data filling functions, the second data filling functions comprising a plurality of sub-functions, the second data filling functions being used to determine a sub-function from the plurality of sub-functions and run, wherein, the plurality of sub-functions comprises a first sub-function, the first sub-function being used to obtain a named entity other than the named entity included in the vocabulary to form the first corpus, and the seed template is further used to determine the first corpus according to the named entity obtained by the first sub-function, and / or the plurality of sub-functions comprises a second sub-function, the second sub-function being used to obtain a word other than a named entity to form the first corpus, and the seed template is further used to determine the first corpus according to the word; determining a training data set according to the first corpus and the annotation information; training the named entity recognition model based on the training data set to obtain a trained named entity recognition model; The method further comprises: in the process of retraining the named entity recognition model subsequently, determining a new seed template for determining a training data set for training the subsequent named entity recognition model by the following steps: obtaining a fourth corpus comprising the annotation information; determining a new first data filling function matched with a named entity corresponding to the annotation information according to the annotation information of the fourth corpus; determining a word other than a named entity in the fourth corpus; determining a new second data filling function based on a regular expression matching method according to the word other than the named entity in the fourth corpus; determining a new seed template according to the new first data filling function and the new second data filling function.

2. The method of claim 1, wherein, The named entity category names stored in the vocabulary comprise the named entity tag, and the first data filling function is used to obtain, according to the named entity tag, a named entity corresponding to the named entity tag to form a first corpus, which comprises: The first data filling function is used to associate the named entity tag and the vocabulary and obtain a named entity corresponding to the named entity tag in the vocabulary.

3. The method of claim 1, wherein, The method further comprises: obtaining seed data, the seed data being data obtained after the original corpus is manually annotated with the named entity tag, the seed data comprising the annotation information; determine a second corpus according to the annotation information of the seed data and the vocabulary; determine the training data set according to the second corpus and the annotation information.

4. The method of claim 1, wherein, The method further comprises: obtain a historical data set, the historical data set comprising a data set used by a trained named entity recognition model and metadata of the used data set, the metadata comprising one or more named entity labels in the data set; obtain a same-label data set in the historical data set according to the metadata, the named entity label of the same-label data set being the same as the named entity label of the first corpus, the same-label data set comprising the annotation information; determine a third corpus according to the annotation information of the same-label data set and the vocabulary; determine the training data set according to the third corpus and the annotation information.

5. The method of claim 1, wherein, The method further comprises: perform model verification on the trained named entity recognition model; update the seed template according to the result of the model verification; determine a new training data set according to the updated seed template; train the named entity recognition model based on the new training data set to obtain a new trained named entity recognition model. 6.A device for training a named entity recognition model, comprising: comprise: a vocabulary obtaining module, configured to obtain a vocabulary, the vocabulary being used to store different named entity category names and named entities corresponding to each named entity category name; a seed template obtaining module, configured to obtain a seed template, the seed template being an abstract semantic representation of a corpus used to train a named entity recognition model, the seed template comprising one or more first data filling functions, the first data filling function being used to obtain a named entity corresponding to a named entity label according to the named entity label, the seed template being used to determine the first corpus according to the named entity and determine annotation information of the first corpus, the annotation information comprising a position of the named entity and a named entity label corresponding to the named entity, the named entity label being used as an expected output of the named entity recognition model to train the model; the seed template further comprises one or more second data filling functions, the second data filling function comprising a plurality of sub-functions, the second data filling function being used to determine a sub-function from the plurality of sub-functions and run the sub-function, wherein the plurality of sub-functions comprise a first sub-function, the first sub-function being used to obtain a named entity other than the named entity included in the vocabulary to constitute the first corpus, the seed template being further used to determine the first corpus according to the named entity obtained by the first sub-function, and / or the plurality of sub-functions comprise a second sub-function, the second sub-function being used to obtain a word other than a named entity to constitute the first corpus, the seed template being further used to determine the first corpus according to the word; a training data set determining module, configured to determine a training data set according to the first corpus and the annotation information; a model training module, configured to train the named entity recognition model based on the training data set to obtain a trained named entity recognition model; The device further comprises a corpus determining module and a data filling function determining module, wherein the corpus determining module is configured to obtain a fourth corpus comprising the annotation information in a subsequent retraining process of the named entity recognition model; The data filling function determining module is configured to determine, according to the annotation information of the fourth corpus, a new first data filling function matched with the named entity corresponding to the annotation information; The data filling function determining module is further configured to determine word segmentation other than the named entity in the fourth corpus, and determine a new second data filling function based on a regular expression matching method according to the word segmentation other than the named entity in the fourth corpus; The seed template obtaining module is further configured to determine a new seed template according to the new first data filling function and the new second data filling function, and the new seed template is used to determine a training data set for subsequent training of the named entity recognition model.

Images