Method for training entity type recognition model, entity type recognition method and device

Abstract

The embodiment of the application discloses a kind of methods for training entity type identification model, entity type identification method and device, it is related to deep learning technical field.The entity type identification method includes: obtaining text to be identified;Entity word in text to be identified is used to generate prompt text;Input text sequence is obtained using text to be identified and prompt text and input entity type identification model, entity type identification model includes prediction network and the encoding network based on pre-training language model, wherein, the feature representation of each element in input text sequence is extracted by encoding network;Prediction network uses the feature representation of each element, predicts the probability distribution of the content masked in prompt text on each word in vocabulary, obtains the entity type corresponding to the entity word in text to be identified according to probability distribution, and vocabulary includes the label word corresponding to each entity type.The application can improve the recognition effect of entity type identification model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, and in particular to a method for training an entity type identification model, an entity type identification method and device. BACKGROUND

[0002] Entity type identification is an important task in information extraction in the field of natural language processing, and its purpose is to classify entities in text into corresponding entity types. It is a basic tool for information extraction, question and answer systems, syntax analysis, machine translation and many other applications. Entity categories can be names, organization names, place names, time, date, organization names, etc.

[0003] In entity type identification, fine-grained type identification is often difficult. Currently, fine-grained entity type identification is usually obtained by adding a classification network to fine-tune a pre-trained language model. However, the entity type identification model trained in this way has poor recognition effect in a small sample scenario. SUMMARY

[0004] Therefore, the present application provides a method for training an entity type identification model, an entity type identification method and device to improve the recognition effect of the entity type identification model.

[0005] The present application provides the following solutions:

[0006] In a first aspect, a method for training an entity type identification model is provided, and the method comprises:

[0007] Obtaining training data comprising a plurality of training samples, wherein each training sample comprises a text sample and an entity type label annotated for an entity word in the text sample;

[0008] Training an entity type identification model using the training data, wherein the entity type identification model comprises a prediction network and an encoding network based on a pre-trained language model; the training comprises:

[0009] The entity word in the text sample is used to generate prompt text containing the entity word in the text sample and masked content; the text sample and the prompt text are used to obtain an input text sequence, the text sequence is input into the encoding network, and a feature representation of each element in the input text sequence is extracted by the encoding network; the prediction network uses the feature representation of each element to predict a probability distribution of each word in the word table for the masked content in the prompt text, and an entity type corresponding to the entity word in the text sample is obtained according to the probability distribution, and the word table includes a label word corresponding to each entity type; the training target includes: minimizing the difference between the entity type corresponding to the entity word in the text sample obtained by the encoding network and the entity type label annotated for the entity word in the text sample.

[0010] According to an implementable manner in the embodiments of the present application, the prompt text is generated by using the entity word in the text sample, including:

[0011] The entity word in the text sample is filled into a corresponding slot in a preset prompt template to obtain the prompt text, and the prompt template contains a slot corresponding to the entity word and masked content identification.

[0012] According to an implementable manner in the embodiments of the present application, the preset prompt template further includes a context reflecting the semantic relationship between the slot corresponding to the entity word and the masked content identification; or,

[0013] The preset prompt template further includes a parameterized context of the slot corresponding to the entity word and the masked content identification, and the parameterized context is updated in the training process of the entity type identification model.

[0014] According to an implementable manner in the embodiments of the present application, the word table further includes synonyms of the label words corresponding to each entity type.

[0015] According to the probability distribution, the entity type corresponding to the entity word in the text sample is obtained, including:

[0016] For each entity type, a probability statistical value of each word in the word table for the masked content in the prompt text is determined respectively, and the entity type with the maximum corresponding probability statistical value is determined as the entity type corresponding to the entity word in the text sample.

[0017] In a second aspect, a method for entity type identification is provided, including:

[0018] Obtaining a text to be identified;

[0019] The entity word in the to-be-recognized text is used to generate a prompt text, and the prompt text includes the entity word in the to-be-recognized text and masked content;

[0020] The to-be-recognized text and the prompt text are used to obtain an input text sequence, the input text sequence is input into an entity type recognition model, and an entity type predicted by the entity type recognition model for the entity word in the to-be-recognized text is acquired, the entity type recognition model includes a prediction network and an encoding network based on a pre-trained language model;

[0021] The encoding network extracts a feature representation of each element in the input text sequence; the prediction network uses the feature representation of each element to predict a probability distribution of each word in a word table for the masked content in the prompt text, and an entity type corresponding to the entity word in the to-be-recognized text is obtained according to the probability distribution, the word table includes a label word corresponding to each entity type.

[0022] According to an implementable manner in the embodiments of the present application, the entity word in the to-be-recognized text is used to generate a prompt text, including:

[0023] The entity word in the to-be-recognized text is filled into a corresponding slot in a preset prompt template to obtain a prompt text, and the preset template includes a slot corresponding to the entity word and masked content.

[0024] According to an implementable manner in the embodiments of the present application, the word table further includes a synonym of the label word corresponding to each entity type.

[0025] According to the probability distribution, the entity type corresponding to the entity word in the to-be-recognized text is obtained, including:

[0026] For each entity type, a probability statistical value of the masked content in the prompt text on the word corresponding to each entity type in the word table is determined respectively, and the entity type with the maximum corresponding probability statistical value is determined as the entity type corresponding to the entity word in the to-be-recognized text.

[0027] According to an implementable manner in the embodiments of the present application, each text in a preset text set is taken as the to-be-recognized text, and a knowledge graph is constructed using the entity word in the to-be-recognized text and the entity type corresponding to the entity word; or,

[0028] The text input by a user in an intelligent customer service system is taken as the to-be-recognized text, and the entity word in the to-be-recognized text and the entity type corresponding to the entity word are used to match attribute information or a service item corresponding to the entity word in a knowledge base corresponding to the entity type, so as to return to the user.

[0029] In a third aspect, a device for training an entity type recognition model is provided, and the device includes:

[0030] a sample obtaining unit configured to obtain training data comprising a plurality of training samples, each of the training samples comprising a text sample and an entity type label annotated for an entity word in the text sample;

[0031] a model training unit configured to train an entity type identification model using the training data, the entity type identification model comprising a prediction network and an encoding network based on a pre-trained language model; the training comprising: generating a prompt text using the entity word in the text sample, the prompt text containing the entity word in the text sample and a masked content; obtaining an input text sequence using the text sample and the prompt text, inputting the input text sequence into the encoding network, and extracting feature representations of elements in the input text sequence by the encoding network; the prediction network predicting a probability distribution of each word in a word table for the masked content in the prompt text using the feature representations of the elements, obtaining an entity type corresponding to the entity word in the text sample according to the probability distribution, the word table comprising label words corresponding to each entity type; and the training target comprising minimizing a difference between the entity type corresponding to the entity word in the text sample obtained by the encoding network and the entity type label annotated for the entity word in the text sample.

[0032] In a fourth aspect, an entity type identification apparatus is provided, and the apparatus comprises:

[0033] a text obtaining unit configured to obtain a text to be identified;

[0034] a prompt generating unit configured to generate a prompt text using an entity word in the text to be identified, the prompt text containing the entity word in the text to be identified and a masked content;

[0035] a type identifying unit configured to obtain an entity type predicted by an entity type identification model for an entity word in the text to be identified by inputting an input text sequence comprising the text to be identified and the prompt text into the entity type identification model, the entity type identification model comprising a prediction network and an encoding network based on a pre-trained language model;

[0036] wherein the encoding network extracts feature representations of elements in the input text sequence; and the prediction network predicts a probability distribution of each word in a word table for the masked content in the prompt text using the feature representations of the elements, and obtains an entity type corresponding to the entity word in the text to be identified according to the probability distribution, the word table comprising label words corresponding to each entity type.

[0037] According to a fifth aspect, a computer readable storage medium is provided, having stored thereon a computer program which, when executed by a processor, implements the steps of the method according to any one of the preceding first aspect.

[0038] According to a sixth aspect, an electronic device is provided, comprising:

[0039] one or more processors; and

[0040] a memory associated with the one or more processors, the memory for storing program instructions which, when read and executed by the one or more processors, perform the steps of the method according to any one of the preceding first aspect.

[0041] According to the embodiments provided in the present application, the following technical effects are disclosed:

[0042] 1) The present application generates prompt text using entity words in a text sample, converts the entity classification task into a mask prediction task of the prompt text, so as to better utilize the understanding ability that the pre-training language model is good at, and even in a small sample scene, accurate identification of fine-grained entity types can be performed, and the identification effect of entity type identification is improved.

[0043] 2) The present application generates prompt text by using a prompt template containing an entity word corresponding slot and a content identifier which is masked, to construct a mask prediction task. The prompt template can use a parameterized context and be updated in the training process of the entity type identification model, so as to automatically complete the construction of the prompt template to achieve better prompting effect.

[0044] 3) The present application generates a word table by expanding the label words corresponding to each entity type, uses the probability statistical value of the masked content on the label words corresponding to each entity type, and determines the entity type corresponding to the entity word according to the probability statistical value, so as to further improve the identification accuracy of the entity type.

[0045] Of course, implementing any product of the present application does not necessarily need to achieve all the advantages described above at the same time. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0047] Figure 1 is the system architecture suitable for the embodiments of the present application;

[0048] Figure 2 A method flowchart for training an entity type recognition model is provided in the embodiments of the present application.

[0049] Figure 3 A principle structure diagram of an entity type recognition model is provided in the embodiments of the present application.

[0050] Figure 4 A method flowchart for entity type recognition is provided in the embodiments of the present application.

[0051] Figure 5 A principle structure diagram of an entity type recognition model is provided in the embodiments of the present application.

[0052] Figure 6 A schematic block diagram of an apparatus for training an entity type recognition model is provided in the embodiments of the present application.

[0053] Figure 7 A schematic block diagram of an apparatus for entity type recognition is provided in the embodiments of the present application.

[0054] Figure 8 A schematic block diagram of an electronic device is provided in the embodiments of the present application. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art belong to the scope of protection of the present application.

[0056] The terms used in the embodiments of the present application are merely for the purpose of describing the specific embodiments, and are not intended to limit the present application. The singular forms "a", "an" and "the" used in the embodiments of the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0057] It should be understood that the term "and / or" used herein is merely to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " in this paper generally represents that the front and rear associated objects are a "or" relationship.

[0058] Depending on context, the word "if" as used herein can be interpreted to mean "when" or "while" or "in response to determining" or "in response to detecting." Similarly, the phrase "if it is determined" or "if [a stated condition or event] is detected" can be interpreted to mean "when it is determined" or "in response to determining" or "when [a stated condition or event] is detected" or "in response to detecting [a stated condition or event]."

[0059] It has been mentioned in the background that a conventional entity type recognition model is usually obtained by adding a classification network to a pre-trained language model for fine-tuning after obtaining a text sample and a type label annotated for an entity in the text sample. The core is to map the feature representation of the entity word to the entity label by the classification network. However, this way faces the problem of poor generalization in the scene of few samples for fine-grained entity type recognition, resulting in poor recognition effect. For example, for the text "London band released an album", the entity "London" needs to be classified into the entity category of "band", and cannot be classified into the entity category of "place name". For such fine-grained entity type recognition, the training method of the conventional entity type recognition model is difficult to accurately recognize.

[0060] In view of this, a new idea is proposed in the present application, that is, a prompt learning mechanism is introduced to train the entity type recognition model. In order to facilitate the understanding of the present application, first, the system architecture applicable to the present application is simply described. Figure 1 An exemplary system architecture to which embodiments of the present application can be applied is shown, which includes a model training device for training an entity type recognition model in an offline manner, and an entity type recognition device for recognizing the entity type of an entity in a text to be recognized online.

[0061] The model training device can train the model by using the method provided in the embodiments of the present application after obtaining the training data, and obtain the entity type recognition model.

[0062] The entity type recognition device uses the established entity type recognition model to recognize the entity type of an entity in a text to be recognized to obtain a recognition result. That is, from the text to be recognized, the entity type of a word that has been annotated as an entity is recognized.

[0063] The model training apparatus and the entity type recognition apparatus can be respectively arranged as independent servers, or arranged in a same server or server group, or arranged in independent or same cloud servers. The cloud server is also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, to solve the defects of large management difficulty and weak service scalability in traditional physical hosts and virtual private server (VPs, Virtual Private Server) services. The model training apparatus and the entity type recognition apparatus can also be arranged in a computer terminal with strong computing capability.

[0064] It should be noted that, in addition to online entity type recognition, the entity type recognition apparatus can also perform entity type recognition in an offline manner, for example, performing entity type recognition on batches of to-be-recognized texts respectively.

[0065] It should be understood that, Figure 1 The number of model training apparatuses, entity type recognition apparatuses and entity type recognition models in the system is only illustrative. According to the implementation needs, there can be any number of model training apparatuses, entity type recognition apparatuses and entity type recognition models.

[0066] Figure 2 A method flowchart for training an entity type recognition model is provided for the embodiments of the present application, and the method flowchart can be executed by the model training apparatus in the system shown in Figure 1 As shown in the method shown in Figure 2 The method can include the following steps:

[0067] Step 202: Obtain training data including a plurality of training samples, and the training sample includes a text sample and an entity type label annotated for an entity word in the text sample.

[0068] Step 204: Train the entity type recognition model using the training data, and the entity type recognition model includes a prediction network and an encoding network based on a pre-trained language model; the training includes: generating a prompt text using the entity word in the text sample, the prompt text containing the entity word in the text sample and the masked content; obtaining an input text sequence using the text sample and the prompt text and inputting the encoding network, and extracting feature representations of each element in the input text sequence by the encoding network; the prediction network uses the feature representations of each element to predict the probability distribution of the masked content in the prompt text on each word in the word table, and obtains the entity type corresponding to the entity word in the text sample according to the probability distribution, and the word table includes a label word corresponding to each entity type; the training target includes: minimizing the difference between the entity type corresponding to the entity word in the text sample obtained by the encoding network and the entity type label annotated for the entity word in the text sample.

[0069] As can be seen from the above process, the entity word in the text sample is used to generate the prompt text, the entity classification task is converted into the mask prediction task of the prompt text, so that the understanding ability of the pre-training language model is better utilized, and even in the small sample scene, the fine-grained entity type can be accurately identified, and the identification effect of the entity type identification is improved.

[0070] The steps in the above process will be described in detail below.

[0071] First, the step 202, i.e., "obtaining training data including a plurality of training samples", will be described in detail in combination with an embodiment.

[0072] The training sample used in the embodiment of the present application is mainly a text sample containing entity words. The entity words in the text sample are pre-labeled, and the entity type labels for the entity words in the text sample are also pre-labeled. The entity type label can be a type label of various granularities. A relatively coarse-grained type label can be used, such as a person name, a place name, a time, a date, an organization name, a currency, a movie name, a book name, etc. A relatively fine-grained type label can also be used, such as a person name being further divided into an actor name, a writer name, an entrepreneur name, etc. A place name is further divided into a country name, a city name, a district name, etc. Assuming that the full set of entity labels Y = {y1, y2, …, ym} to be predicted in the embodiment of the present application, where m is the total number of entity type labels. m}, wherein m is the total number of entity type labels.

[0073] The step 204, i.e., "training an entity type identification model using the training data", will be described in detail below.

[0074] Figure 3 The structure diagram of the entity type identification model provided in the embodiment of the present application is shown in FIG. 1, which mainly includes an encoding network and a prediction network. Figure 3

[0075] The encoding network is implemented based on a pre-training language model. The pre-training language model used can be, for example, a BERT (Bidirectional Encoder Representation from Transformers) model, an XLNet model, a GPT (Generative Pre-Training) model, a T5 model, etc. It can also be an extended model based on the above models, such as a RoBERTa (ARobustly Optimized BERT Approach) model, a GPT-2 model, a GPT-3 model, etc.

[0076] ​The core of the present application is to convert the entity classification task into a mask prediction task through the construction of a prompt text. Therefore, in the embodiment of the present application, the input of the encoding network will input the prompt text generated by using the entity words in the text sample in addition to the text sample. The prompt text contains entity words in the text sample and the masked content.

[0077] When generating the prompt text, a preset prompt template can be used. The entity words in the text sample are filled into the corresponding slot in the preset prompt template to obtain the prompt text. The prompt template contains the slot corresponding to the entity word and the masked content identifier.

[0078] As one of the implementable modes, the above-mentioned prompt template can be a template set by artificial experience. In addition to the entity word and the masked content, the prompt template also contains the context of the entity word and the masked content in the text sample to reflect the semantic relationship between the entity word and the masked content. These contexts can be set by artificial experience.

[0079] For example, a prompt template such as "

entity word

mask

entity word

mask

mask

[0080] For example, a prompt template such as "The type of

entity word

mask

entity word

mask

mask

[0081] The above-mentioned prompt template is used to express the semantic relationship between

entity word

mask

mask

[0082] As another implementable mode, the above-mentioned prompt template can be a parameterized template, that is, a "soft template". In addition to the entity word and the masked content, the prompt template also contains the parameterized context of the slot corresponding to the entity word and the identifier of the masked content. The parameterized context is no longer a fixed text, but a parameter that is gradually updated in the training process of the entity type identification model, that is, these parameterized contexts are learned in the model training process.

[0083] For example, a prompt template such as "

entity word

P1

P2

Pn

mask

entity word

mask

P1

P2

Pn

P1

P2

Pn

mask

[0084] The text sample and the prompt text are input into the encoding network as an input text sequence. For example, the text sample and the prompt text can be spliced to obtain the input text sequence. The input text sequence usually starts with a start symbol (usually represented by "[CLS]"), and each Token (element) after the start symbol usually includes characters (some languages may use words) and a separator (usually represented by "[SEP]"). Assuming that the input text sequence contains M Tokens, before the encoding network, the Embedding of each Token can be processed first by using an embedding network, and then the Embedding of each Token is encoded by using each Transformer layer in the pre-trained language model to obtain the feature representation of each Token.

[0085] Among them, the Embedding of each Token at least includes word Embedding and position Embedding. The word Embedding is to encode each Token into a word vector to obtain a word vector representation. The position Embedding is to encode the position of each Token in the input text sequence to obtain a position representation. For example, the positions of each Token in the input text sequence are numbered as 0, 1, 2, 3, 4, 5, and 6 in turn.

[0086] The encoding network is used to extract the feature representation of each Token in the input text sequence, and then output to the prediction network. The prediction network uses the feature representation of each Token to predict the probability distribution of the masked content in the prompt text on each word in the vocabulary, and obtains the entity type corresponding to the entity word in the text sample according to the probability distribution.

[0087] As one of the implementable ways, the above-mentioned vocabulary can include a label word corresponding to each entity type, that is, a label used when labeling the entity word.

[0088] As another preferred embodiment, the above-mentioned vocabulary can be constructed in advance by using the label word corresponding to each entity type and its synonym. Then for each entity label Y iFor any entity category tag "city," its synonyms can be used to expand the vocabulary. For example, for the entity category tag "city," its synonyms such as "city," "town," and "metropolis" can be added to the vocabulary. This method is equivalent to expanding the tag words to obtain the vocabulary; that is, mapping the tag set Y to a word set V, which is the vocabulary, represented as V = {w1, w2, ..., w...}. n}

[0089] A prediction network can essentially be a classification network that uses the feature representations of each element to map the masked content in the prompt text to each word in the vocabulary, thereby obtaining the probability value of the masked content on each word in the vocabulary.

[0090] If the vocabulary only contains entity tags corresponding to each entity type, then the probability value of the masked content on each word in the vocabulary is actually the probability value of the entity word in the text sample on the corresponding entity type tag word. The entity type with the highest probability value is selected as the entity type of the entity word in the text sample.

[0091] If the vocabulary contains tag words and synonyms corresponding to each entity type, then for each entity type, the probability statistics of the masked content in the prompt text on the corresponding words of each entity type in the vocabulary can be determined separately. The entity type with the highest probability statistics is then identified as the entity type corresponding to the entity word in the text sample. The probability statistics can be the maximum probability, average probability, summation, etc.

[0092] For example, suppose there is an entity type T in the vocabulary. i The tag y i And its corresponding synonyms A1, A2 and A3, to locate the entity words in the text sample in y i The average of the probability values ​​on A1, A2, and A3 is used as the entity word in the text sample of type y. i The probability value on the [text] is [the probability value]. If, among all categories, the entity word in this text sample is of type T... i If the probability value is the highest, then the entity type corresponding to the entity word is determined to be T. i .

[0093] This is equivalent to transforming the prediction of entity types for entity words into the prediction of masked content in the prompt text, which can be represented as:

[0094]

[0095] P(y i x) indicates that the entity word x is in the tag y i The probability value of w j For label y iThe corresponding word in the glossary. k is y i The number of corresponding words in the glossary. T(x) represents the prompt text.

[0096] The training target is to minimize the difference between the entity type corresponding to the entity word in the text sample obtained by the encoding network and the entity type label annotated in the entity word in the text sample. In the embodiments of the present application, a loss function can be constructed according to the above training target, and the model parameters are updated by using the value of the loss function in each iteration, such as gradient descent, until the preset training end condition is met. The training end condition can include, for example, the value of the loss function is less than or equal to the preset loss function threshold, the number of iterations reaches the preset number threshold, and the like.

[0097] When updating the model parameters, in addition to updating the parameters of the encoding network and the prediction network, the parameters of the embedding network can also be further updated. In addition, the parameters of the embedding network can not participate in the update, but the pre-trained parameter value remains unchanged.

[0098] If the prompt template uses a parameterized context, the parameters of the prompt template will also be updated during model training. If the model parameters are represented as θ, and the parameters of the prompt template are represented as The cross-entropy loss function Loss can be constructed, represented as:

[0099]

[0100] After training, an entity type recognition model is obtained, and based on the entity type recognition model, entity type recognition can be performed. Figure 4 The method flow of the entity type recognition provided in the embodiments of the present application can be executed by the entity type recognition device in the system architecture shown in Figure 1 As shown in Figure 4 The method can include:

[0101] Step 402: Obtain the text to be recognized.

[0102] The text to be recognized involved in the embodiments of the present application refers to the text to be recognized containing entity words, and the entity words can be annotated by special symbols. The goal of the present application is to identify the entity type information of the annotated entity words from the text to be recognized.

[0103] According to different application scenarios, the source of the text to be recognized is different. For example, in the knowledge graph construction scenario, the text to be recognized can be derived from a text set containing entity words. For another example, in the customer service scenario, the text to be recognized can be derived from the text input by the user in the intelligent customer service system. Subsequently, the two scenarios will be introduced as examples, and other scenarios will not be enumerated.

[0104] Step 404: generating a prompt text using the entity word in the text to be recognized, the prompt text containing the entity word in the text to be recognized and the masked content.

[0105] In this step, the generation method of the prompt text is consistent with the generation method of the prompt text in the process of training the entity type recognition model. A preset prompt template can be used to fill the entity word in the text to be recognized into the corresponding slot in the preset prompt template to obtain the prompt text. The prompt template contains the slot corresponding to the entity word and the masked content identifier.

[0106] As one of the implementable ways, the above-mentioned prompt template can be a template set by human experience. In addition to the entity word and the masked content, the prompt template also contains the context of the entity word and the masked content in the text sample to reflect the semantic relationship between the entity word and the masked content. These contexts can be set by human experience.

[0107] As another implementable way, if the prompt template is a parameterized template, i.e., a "soft template", in addition to the entity word and the masked content, the prompt template also contains the parameterized context of the slot corresponding to the entity word and the masked content identifier. The parameterized context is learned in the model training process. In this step, the entity word in the text to be recognized is filled into the parameterized template learned by the model training.

[0108] Step 406: obtaining an input text sequence using the text to be recognized and the prompt text, inputting the input text sequence into the entity type recognition model to obtain the entity type predicted by the entity type recognition model for the entity word in the text to be recognized, the entity type recognition model including a prediction network and an encoding network based on a pre-trained language model.

[0109] The encoding network extracts the feature representation of each element in the input text sequence; the prediction network uses the feature representation of each element to predict the probability distribution of the masked content in the prompt text on each word in the word table, and obtains the entity type corresponding to the entity word in the text to be recognized according to the probability distribution, the word table including the label word corresponding to each entity type.

[0110] As Figure 5As shown in the middle, the input text sequence of the to-be-identified text and the prompt text is input into the encoding network, for example, the input text sequence can be obtained by splicing the text sample and the prompt text. The input text sequence usually starts with a start symbol (usually represented by “[CLS]”), and each Token (element) after the start symbol usually includes characters (some languages may use words) and a separator (usually represented by “[SEP]”). Assuming that the input text sequence includes M Tokens, the Embedding (embedding) processing of each Token can be performed by using an embedding network before the encoding network, and then the Embedding of each Token is encoded by using each Transformer layer in the pre-trained language model to obtain the feature representation of each Token.

[0111] The Embedding processing of each Token at least includes word Embedding and position Embedding. The word Embedding is to encode each Token by using a word vector to obtain a word vector representation. The position Embedding is to encode the position of each Token in the input text sequence to obtain a position representation.

[0112] The encoding network is used to extract the feature representation of each Token in the input text sequence, and then outputs the feature representation to the prediction network. The prediction network uses the feature representation of each element to predict the probability distribution of the masked content in the prompt text on each word in the vocabulary, and obtains the entity type corresponding to the entity word in the to-be-identified text according to the probability distribution.

[0113] As one of the implementable manners, the above-mentioned vocabulary can include the label words corresponding to each entity type.

[0114] As another preferred embodiment, the above-mentioned vocabulary can be constructed in advance by using the label words corresponding to each entity type and the synonyms thereof.

[0115] The prediction network uses the feature representation of each element to map the masked content in the prompt text to each word in the vocabulary to obtain the probability value of the masked content on each word in the vocabulary.

[0116] If the vocabulary only includes the entity labels corresponding to each entity type, the probability value of the masked content on each word in the vocabulary is actually the probability value of the entity word in the to-be-identified text on the corresponding entity type label, and the entity type label with the maximum probability value is selected as the entity type of the entity word in the to-be-identified text.

[0117] If the word list contains label words corresponding to each entity type and their synonyms, for each entity type, the probability statistical value of the masked content in the prompt text on the corresponding word of each entity type can be determined respectively, and the entity type corresponding to the entity word in the to-be-identified text is determined as the entity type corresponding to the entity word with the maximum corresponding probability statistical value. The probability statistical value can be the maximum probability, the average value, the sum value, etc.

[0118] Taking the to-be-identified text "London band released an album" as an example, the prompt text "London is a

mask

mask

mask

mask

mask

mask

[0119] The entity type identification model and the entity type identification method trained by the embodiments of the present application can be applied to various application scenarios, of which only two are listed here:

[0120] Application scenario 1: constructing a knowledge graph.

[0121] A knowledge graph is mainly used to describe various entities existing in the real world and their relationships. The entities can be mined from a large amount of text. In the process of building a knowledge graph, in addition to mining entities, the types of entities also need to be identified. And with the continuous emergence of new things and the continuous improvement of user needs, finer granularity division and identification of entity types are needed. However, the sample distribution of different entity types is uneven, and the entity type identification model trained by the conventional classification task is difficult to effectively distinguish the entity types with small sample size. If the classification task can be converted into a mask prediction task of the prompt text by using the method provided in the embodiments of the present application, the understanding ability of the pre-training language model can be fully utilized, and even in the small sample scene, the entity types can be accurately identified, and the identification effect of the entity type identification is improved. Then, the entity type identification model is used to identify the entity types in the text in the preset text set, so as to build a knowledge graph.

[0122] Application scenario 2: customer service.

[0123] Now more and more customers use online customer service to communicate when encountering service problems. Many current online customer service systems use intelligent customer service systems, and the text input by the user in the intelligent customer service system is used as the text to be identified. The entity words extracted from the text to be identified and the determined entity type information are used to match the attribute information or service items corresponding to the entity words in the knowledge base corresponding to the entity type, and the attribute information or service items are returned to the user.

[0124] The above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be executed in an order different than the order in the embodiments and still achieve the desired result. In addition, the processes depicted in the figures do not necessarily require the particular order shown or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous or possible.

[0125] According to an embodiment of another aspect, a device for training an entity type identification model is provided. Figure 6 A schematic block diagram of a device for training an entity type identification model according to an embodiment is shown, which is Figure 1 The model training device in the illustrated architecture. As Figure 6 The device 600 shown includes a sample acquisition unit 601 and a model training unit 602. The main functions of each component unit are as follows:

[0126] The sample acquisition unit 601 is configured to acquire training data including a plurality of training samples, and the training sample includes a text sample and an entity type label annotated in the entity word in the text sample.

[0127] The model training unit 602 is configured to train an entity type identification model by using training data, the entity type identification model comprising a prediction network and an encoding network based on a pre-trained language model; the training comprises: generating prompt text by using an entity word in a text sample, the prompt text containing the entity word in the text sample and masked content; obtaining an input text sequence by using the text sample and the prompt text, inputting the input text sequence into the encoding network, and extracting feature representations of elements in the input text sequence by the encoding network; the prediction network uses the feature representations of the elements to predict probability distributions of the masked content in the prompt text on words in a word table, and obtains an entity type corresponding to the entity word in the text sample according to the probability distributions, the word table comprising label words corresponding to each entity type; the training target comprises minimizing the difference between the entity type corresponding to the entity word in the text sample obtained by the encoding network and an entity type label annotated for the entity word in the text sample.

[0128] As one of the implementable manners, the model training unit 602, when generating the prompt text by using the entity word in the text sample, can be specifically configured to: fill the entity word in the text sample into a corresponding slot in a preset prompt template to obtain the prompt text, the prompt template containing the slot corresponding to the entity word and the masked content identifier.

[0129] As one of the implementable manners, the preset prompt template further comprises a context embodying a semantic relationship between the slot corresponding to the entity word and the masked content identifier, the context being set by a person according to experience.

[0130] As another implementable manner, the preset prompt template further comprises a parameterized context of the slot corresponding to the entity word and the masked content identifier, the parameterized context being updated in the training process of the entity type identification model.

[0131] As one of the implementable manners, the word table is pre-constructed by using the label words corresponding to each entity type and their synonyms.

[0132] As one of the implementable manners, the model training unit 602, when obtaining the entity type corresponding to the entity word in the text sample according to the probability distributions, can be specifically configured to: for each entity type, determine a probability statistical value of the masked content in the prompt text on the word corresponding to each entity type in the word table, and determine the entity type with the largest corresponding probability statistical value as the entity type corresponding to the entity word in the text sample.

[0133] The embodiment of the present application also provides an entity type identification device. Figure 7 A schematic block diagram of an entity type identification device according to one embodiment is shown. As shown in FIG. 2, the entity type identification device comprises a model training unit 202. Figure 7As shown, the apparatus 700 comprises a text acquisition unit 701, a prompt generation unit 702, and a type identification unit 703. The main functions of each component unit are as follows:

[0134] The text acquisition unit 701 is configured to acquire the text to be recognized.

[0135] The prompt generation unit 702 is configured to generate a prompt text using the entity words in the text to be recognized, the prompt text containing the entity words in the text to be recognized and the masked content.

[0136] The type identification unit 703 is configured to obtain an input text sequence using the text to be recognized and the prompt text, input the input text sequence into an entity type identification model, and acquire the entity type predicted by the entity type identification model for the entity words in the text to be recognized, the entity type identification model comprising a prediction network and an encoding network based on a pre-trained language model.

[0137] The encoding network extracts the feature representation of each element in the input text sequence; the prediction network uses the feature representation of each element to predict the probability distribution of the masked content in the prompt text on each word in the word table, and obtains the entity type corresponding to the entity word in the text to be recognized according to the probability distribution, the word table comprising label words corresponding to each entity type.

[0138] As one of the implementable ways, the prompt generation unit 702 fills the entity words in the text to be recognized into the corresponding slot positions in the preset prompt template to obtain the prompt text, the preset template containing slot positions corresponding to entity words and masked content identifiers.

[0139] As one of the implementable ways, the above word table is pre-constructed using the label words corresponding to each entity type and their synonyms.

[0140] As one of the implementable ways, when obtaining the entity type corresponding to the entity word in the text to be recognized according to the probability distribution, the type identification unit 703 can be specifically configured to: for each entity type, determine the probability statistical value of the masked content in the prompt text on the word corresponding to each entity type, and determine the entity type corresponding to the entity word in the text to be recognized as the entity type with the largest corresponding probability statistical value.

[0141] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, they are described more simply, and the relevant parts can be referred to the part of the method embodiments. The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e. they can be located in one place, or distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment. Those skilled in the art can understand and implement it without creative labor.

[0142] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0143] In addition, the embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the steps of the method in any one of the preceding method embodiments.

[0144] And an electronic device, comprising:

[0145] one or more processors; and

[0146] a memory associated with the one or more processors, the memory configured to store program instructions that, when executed by the one or more processors, perform the steps of the method of any one of the preceding method embodiments.

[0147] The present application also provides a computer program product, comprising a computer program, which, when executed by a processor, realizes the steps of the method in any one of the preceding method embodiments.

[0148] wherein, Figure 8An exemplary architecture of the electronic device can include a processor 810, a video display adapter 811, a disk drive 812, an input / output interface 813, a network interface 814, and a memory 820. The processor 810, the video display adapter 811, the disk drive 812, the input / output interface 813, the network interface 814, and the memory 820 can be communicatively connected through a communication bus 830.

[0149] The processor 810 can be implemented in the form of a general-purpose CPU, a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute related programs to implement the technical solutions provided in the present application.

[0150] The memory 820 can be implemented in the form of a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, or the like. The memory 820 can store an operating system 821 for controlling the operation of the electronic device 800, a basic input / output system (BIOS) 822 for controlling the low-level operation of the electronic device 800. In addition, a web browser 823, a data storage management system 824, and a model training apparatus / entity type identification apparatus 825, and the like can also be stored. The model training apparatus / entity type identification apparatus 825 can be an application program for implementing the foregoing steps in the embodiments of the present application. In summary, when the technical solutions provided in the present application are implemented by software or firmware, the related program codes are stored in the memory 820 and executed by the processor 810.

[0151] The input / output interface 813 is configured to connect an input / output module to realize information input and output. The input / output module can be configured as a component in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. The input device can include a keyboard, a mouse, a touch screen, a microphone, various sensors, and the like, and the output device can include a display, a speaker, a vibrator, an indicator, and the like.

[0152] The network interface 814 is configured to connect a communication module (not shown in the figure) to realize the communication interaction between the device and other devices. The communication module can realize communication through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.).

[0153] Bus 830 includes a path for transferring information between the various components of the device (e.g., processor 810, video display adapter 811, disk drive 812, input / output interface 813, network interface 814, and memory 820).

[0154] It should be noted that although the above device only shows the processor 810, video display adapter 811, disk drive 812, input / output interface 813, network interface 814, memory 820, bus 830, etc., but in the process of implementation, the device can also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device can also contain only the components necessary to implement the scheme of the present application, and does not have to contain all the components shown in the figure.

[0155] From the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and the necessary general hardware platform. Based on such understanding, the technical solutions of the present application can be embodied in the form of a computer program product, which can be stored in a storage medium such as ROM / RAM, disk, optical disk, etc., including a number of instructions for making a computer device (which can be a personal computer, server, or network device, etc.) execute the methods described in various embodiments or some parts of the embodiments of the present application.

[0156] The above describes the technical solutions provided by the present application in detail, and the principles and implementation manners of the present application are described by applying specific examples; the above description of the embodiments is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range will be changed. In conclusion, the content of the present description should not be understood as a limitation of the present application.

Claims

1. A method for training an entity type recognition model, characterized in that, The method includes: Acquire training data including multiple training samples, wherein the training samples include text samples and entity type labels of entity words in the text samples; The entity type recognition model is trained using the training data. The entity type recognition model includes a prediction network and an encoding network based on a pre-trained language model. The training includes: The system generates prompt text using entity words from text samples. The prompt text includes entity words from the text samples and masked content. An input text sequence is obtained using the text samples and the prompt text. This input text sequence is then fed into an encoding network, which extracts feature representations of each element in the input text sequence. A prediction network uses these feature representations to predict the probability distribution of the masked content in the prompt text across words in a vocabulary. Based on this probability distribution, the system obtains the entity types corresponding to the entity words in the text samples. The vocabulary includes tag words corresponding to each entity type. The training objective is to minimize the difference between the entity types corresponding to the entity words in the text samples obtained by the encoding network and the entity type tags labeled on the entity words in the text samples. The feature representation includes word vector representation and position representation. The step of extracting the feature representation of each element in the input text sequence by the encoding network includes: performing word vector encoding on the element to obtain the word vector representation of the element, and encoding the position of the element in the input text sequence to obtain the position representation of the element. The vocabulary is pre-constructed using the tag words corresponding to each entity type and the synonyms of the tag words.

2. The method according to claim 1, characterized in that, The method of generating prompt text using entity words in text samples includes: The entity words in the text sample are filled into the corresponding slots in the preset prompt template to obtain the prompt text. The prompt template includes the slots corresponding to the entity words and the masked content identifiers.

3. The method according to claim 2, characterized in that, The preset prompt template also includes context reflecting the semantic relationship between the slot corresponding to the entity word and the masked content identifier; or, The preset prompt template also includes the slot corresponding to the entity word and the parameterized context of the masked content identifier, and the parameterized context is updated during the training of the entity type recognition model.

4. The method according to any one of claims 1 to 3, characterized in that, The vocabulary also includes synonyms for the tag words corresponding to each entity type; Based on the probability distribution, the entity types corresponding to the entity words in the text sample include: For each entity type, the probability statistics of the masked content in the prompt text are determined for each word corresponding to each entity type in the vocabulary. The entity type with the highest probability statistics is determined as the entity type corresponding to the entity word in the text sample.

5. A method for entity type recognition, characterized in that, The method includes: Obtain the text to be recognized; A prompt text is generated using entity words in the text to be identified. The prompt text contains entity words in the text to be identified and masked content. An input text sequence is obtained using the text to be identified and the prompt text. The input text sequence is then input into an entity type recognition model to obtain the entity type predicted by the entity type recognition model for entity words in the text to be identified. The entity type recognition model includes a prediction network and an encoding network based on a pre-trained language model. The encoding network extracts feature representations of each element in the input text sequence; the prediction network uses the feature representations of each element to predict the probability distribution of the masked content in the prompt text on each word in the vocabulary, and obtains the entity type corresponding to the entity word in the text to be identified based on the probability distribution, wherein the vocabulary includes tag words corresponding to each entity type; The entity type recognition model is trained using the method described in any one of claims 1 to 4.

6. The method according to claim 5, characterized in that, Generating prompt text using entity words in the text to be identified includes: The entity words in the text to be identified are filled into the corresponding slots in the preset prompt template to obtain the prompt text. The preset template includes the slots corresponding to the entity words and the masked content identifiers.

7. The method according to claim 5, characterized in that, The vocabulary also includes synonyms for the tag words corresponding to each entity type; The entity types corresponding to the entity words in the text to be identified, obtained based on the probability distribution, include: For each entity type, the probability statistics of the masked content in the prompt text on the corresponding words of each entity type in the vocabulary are determined, and the entity type with the largest corresponding probability statistics is determined as the entity type corresponding to the entity word in the text to be identified.

8. The method according to any one of claims 5 to 7, characterized in that, Each text in a preset text set is used as the text to be identified, and a knowledge graph is constructed using the entity words in the text to be identified and their corresponding entity types; or, The text entered by the user in the intelligent customer service system is used as the text to be identified. The entity words in the text to be identified and their corresponding entity types are used to match the attribute information or service items corresponding to the entity words in the knowledge base corresponding to the entity type, and then returned to the user.

9. An apparatus for training an entity type recognition model, characterized in that, The device includes: The sample acquisition unit is configured to acquire training data including multiple training samples, wherein the training samples include text samples and entity type labels of entity words in the text samples. The model training unit is configured to train an entity type recognition model using the training data. The entity type recognition model includes a prediction network and an encoding network based on a pre-trained language model. The training includes: generating prompt text using entity words from text samples, the prompt text containing entity words from the text samples and masked content; obtaining an input text sequence using the text samples and prompt text, inputting the input text sequence into the encoding network, and having the encoding network extract feature representations of each element in the input text sequence; the prediction network using the feature representations of each element to predict the probability distribution of the masked content in the prompt text across words in a vocabulary, and obtaining the entity type corresponding to the entity words in the text samples based on the probability distribution, the vocabulary including tag words corresponding to each entity type; the training objective includes: minimizing the difference between the entity type corresponding to the entity words in the text samples obtained by the encoding network and the entity type tags labeled on the entity words in the text samples. The feature representation includes word vector representation and position representation. The model training unit is used to extract the feature representation of each element in the input text sequence by the encoding network through the following steps: encoding the element with word vectors to obtain the word vector representation of the element, and encoding the position of the element in the input text sequence to obtain the position representation of the element. The vocabulary is pre-constructed using the tag words corresponding to each entity type and the synonyms of the tag words.

10. An entity type recognition device, characterized in that, The device includes: The text acquisition unit is configured to acquire the text to be recognized. The prompt generation unit is configured to generate prompt text using entity words in the text to be identified, the prompt text including entity words in the text to be identified and masked content; The type recognition unit is configured to use the text to be recognized and the prompt text as input text sequences, input the input text sequences into an entity type recognition model, and obtain the entity type predicted by the entity type recognition model for entity words in the text to be recognized. The entity type recognition model includes a prediction network and an encoding network based on a pre-trained language model. The encoding network extracts feature representations of each element in the input text sequence; the prediction network uses the feature representations of each element to predict the probability distribution of the masked content in the prompt text on each word in the vocabulary, and obtains the entity type corresponding to the entity word in the text to be identified based on the probability distribution, wherein the vocabulary includes tag words corresponding to each entity type; The entity type recognition model is trained using the method described in any one of claims 1 to 4.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1 to 8.

12. An electronic device, characterized in that, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 8.

Images