Method, device and computer-readable storage medium for entity classification model training

Abstract

The embodiments of this specification provide a training method, device and computer-readable storage medium for an entity classification model. The method includes the following steps. Obtain a first sample set, which includes a first training sample, the first training sample includes a first original text and its corresponding first template text, the first template text is used to describe the first entity in the first original text and multiple first entity types corresponding to the first entity; input the first original text into the first entity classification model to obtain a first prediction result, which includes sequentially outputting the first probability score of each word in the first template text; according to the first prediction result, determine the first prediction loss corresponding to the first training sample; based on the prediction loss corresponding to each training sample in the first sample set, train the first entity classification model so that the entity classification model learns the information of the original text to achieve multi-label classification of entities.

Description

Technical Field

[0001] One or more embodiments of this specification relate to the field of machine learning, and in particular, to methods, devices, and computer-readable storage media for training entity classification models. Background Art

[0002] With the development of deep learning technology, artificial intelligence (AI) models based on deep learning technology have gained increasing attention and are widely used in text entity type recognition, that is, identifying entities in text through AI models, classifying the identified entities, and determining the entity type to which the entities belong.

[0003] Currently, entities in text belong to only one entity type. However, in some possible scenarios, an entity in a text may belong to multiple entity types. The diversity of entity types of a single entity brings challenges to the classification of entities in text. Summary of the Invention

[0004] One or more embodiments of this specification describe a method and apparatus for training an entity classification model, which constructs a template text of an original text (used to illustrate an entity and multiple entity types of the entity in the original text), takes maximizing the probability score of each word in the template text as the goal, and trains the entity classification model so that the entity classification model learns the information of the original text to realize the classification of multiple entity types of the entity.

[0005] According to a first aspect, a method for training an entity classification model is provided, comprising:

[0006] Acquire a first sample set, including a first training sample, wherein the first training sample includes a first original text and a first template text corresponding thereto, wherein the first template text is used to describe a first entity in the first original text and a plurality of first entity types corresponding to the first entity;

[0007] Inputting the first original text into a first entity classification model to obtain a first prediction result, which includes sequentially outputting a first probability score of each word in the first template text;

[0008] Determining a first prediction loss corresponding to the first training sample according to the first prediction result;

[0009] The first entity classification model is trained based on the prediction loss corresponding to each training sample in the first sample set.

[0010] According to a possible implementation, the first training sample further includes a second template text, where the second template text is used to describe non-entities in the first original text that do not belong to the multiple first entity types; the first prediction result further includes sequentially outputting a second probability score for each word in the second template text.

[0011] According to a possible implementation, the first sample set also includes a second training sample, the second training sample includes a second original text and a corresponding third template text, the third template text is used to describe the second entity in the second original text and a single second entity type corresponding to the second entity; the method also includes: inputting the second original text into the first entity classification model to obtain a second prediction result, which includes a second probability score of each word in the third template text output sequentially; and determining a second prediction loss corresponding to the second training sample based at least on the second prediction result.

[0012] In one embodiment, the second training sample also includes a fourth template text, which is used to describe non-entities other than the second entity type in the second original text; the second prediction result also includes a fourth probability score of each word in the fourth template text output sequentially.

[0013] In one embodiment, the plurality of first entity types includes the second entity type.

[0014] In one embodiment, the method also includes: obtaining a single-type template, the single-type template including an entity slot and an entity type slot; filling the second entity in the second original text and its corresponding single first entity type into the entity slot and entity type slot in the single-type template respectively to obtain the third template text.

[0015] According to a feasible implementation method, the method also includes: obtaining a multi-type template corresponding to the first original text, the multi-type template including one entity slot and multiple entity type slots; filling the first entity in the first original text and its corresponding multiple first entity types into one entity slot and multiple entity type slots in the multi-type template to obtain the first template sample; the number of the multiple entity type slots is the same as the number of the multiple first entity types.

[0016] According to a feasible implementation, the method further includes: obtaining a second sample set, which includes multiple third training samples, each third training sample includes a third original text and a corresponding fifth template text, and the fifth template text is used to describe the third entity in the corresponding third original text and a single third entity type corresponding to the third entity; inputting the third original text into a second entity classification model to obtain a third prediction result, which includes sequentially outputting a fifth probability score of each word in the fifth template text; training the second entity classification model based on the third prediction results corresponding to each third training sample in the second sample set, and using the trained second entity classification model as the first entity classification model.

[0017] In one embodiment, the third entity type is any one of the multiple first entity types; and the number of samples in the second sample set is greater than the number of samples in the first sample set.

[0018] According to a feasible implementation manner, the first entity classification model includes an encoder and a decoder; the first original text is input into the first entity classification model to obtain a first prediction result, including: using the encoder to encode the first original text to obtain an encoding vector; using the decoder to decode the encoding vector and output a word probability distribution corresponding to each of multiple moments; for each target moment among the multiple moments, determining the probability of a target word from the word probability distribution corresponding to the target moment, and incorporating it into the first probability score, the target word being the word in the first template text whose number corresponds to the target moment.

[0019] According to a feasible implementation manner, the multiple first entity types include mobile phone numbers and application accounts.

[0020] According to a second aspect, a training device for an entity classification model is provided, comprising:

[0021] a sample set acquisition module configured to acquire a first sample set, including a first training sample, wherein the first training sample includes a first original text and a first template text corresponding thereto, wherein the first template text is used to describe a first entity in the first original text and a plurality of first entity types corresponding to the first entity;

[0022] a prediction module configured to input the first original text into a first entity classification model to obtain a first prediction result, including sequentially outputting a first probability score of each word in the first template text;

[0023] a loss determination module, configured to determine a first prediction loss corresponding to the first training sample based on the first prediction result;

[0024] The training module is configured to train the first entity classification model based on the prediction loss corresponding to each training sample in the first sample set.

[0025] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method described in the first aspect.

[0026] According to a fourth aspect, a computing device is provided, comprising a memory and a processor, wherein executable code is stored in the memory, and when the processor executes the executable code, the method described in the first aspect is implemented.

[0027] In an embodiment of the present specification, a template text of the original text (used to illustrate an entity in the original text and multiple entity types of the entity) is constructed, and an entity classification model is trained with the goal of maximizing the probability score of each word in the template text, so that the entity classification model learns the information of the original text to realize the classification of multiple entity types of the entity. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0029] Figure 1 A schematic diagram illustrating entity classification by an entity classification model in one embodiment;

[0030] Figure 2 A schematic diagram illustrating a flow chart of a solution for training an entity classification model in one embodiment;

[0031] Figure 3 A schematic diagram illustrating a method for determining a probability distribution of a template text in one embodiment;

[0032] Figure 4 A schematic flow chart showing a method for training an entity classification model in one embodiment is shown;

[0033] Figure 5 A schematic structural diagram of an apparatus for training an entity classification model according to one embodiment is shown. DETAILED DESCRIPTION

[0034] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0035] Entity extraction, also known as named entity recognition (NER), is mainly responsible for identifying the textual scope of entities in text and classifying them into predefined entity categories. Entities can include people, places, organizations, dates, currencies, percentages, etc. It is the basis of question-answering systems, translation systems, and knowledge graphs.

[0036] Currently, an entity classification model is used to identify entities in text and classify the entity into one of multiple preset entity categories. Figure 1 A schematic diagram showing entity classification by an entity classification model in one scenario is shown. Figure 1 As shown in the figure, the entity classification model includes an encoder and a CRF (Conditional Random Fields) layer. In practical applications, the vectorized representation vectors w1, w2, ..., wM of each word in the text are determined; w1, w2, ..., wM are input into the encoder to obtain the classification probability distribution P1, P2, ..., P of w1, w2, ..., wM respectively. M Here, the classification probability distribution includes the probability values ​​of multiple preset categories 0, A, B, ..., N, where 0 represents other categories, usually non-entity categories; finally, through M outputs, the categories corresponding to w1, w2, w3, ..., wM are obtained, and the characters corresponding to consecutive identical categories (excluding category 0) are combined as entities. Among them, the representation vector of a character can be understood as the vector obtained by initially encoding the character, which facilitates the quantification of the character information. In one example, the representation vector at least integrates token_embedding (word embedding) and position embedding (position embedding). Among them, for any character in the text, the character vector table (including the encoding vectors of a large number of characters) is searched, and the encoding vector of the matching character is the token_embedding of the character; based on the sequence number of the character (obtained by sorting the characters in the text), the sorted bit vector table (including the encoding vectors of multiple sequence numbers) is searched, and the encoding vector of the matching sequence number is the position embedding of the character.

[0037] However, in some possible scenarios, an entity in a large amount of text used to train the entity classification model belongs to one entity category, but an entity in a small number of samples belongs to multiple entity categories. One entity category is used as a label, resulting in the problem of small samples in multi-label classification. In addition, since one entity category among the multiple entity categories to which an entity belongs may be dominant, it is difficult to predict the other entity categories. However, these entity categories may be of great significance. Therefore, it is necessary to let the entity classification model learn from the information of small samples to achieve multi-label classification. However, the output of the above entity classification model is a single category for each word in the text, and it is impossible to predict the multiple entity types of an entity. Therefore, the problem of small samples in multi-label classification has become an urgent problem to be solved.

[0038] For example, the small sample for multi-label classification can be a case transcript. To facilitate understanding of the problem of small samples for multi-label classification, the following is a detailed description using a case transcript as an example.

[0039] The process of electronically entering case data into the system relies entirely on manual labor. After recording a case on paper, a case transcript is generated, and staff members enter each element (also understood as an entity) of the case in the transcript into a form. This presents two problems: 1) Manual entry can lead to errors such as typos; 2) Manual entry is inefficient for massive amounts of case transcripts, as dozens of elements for each case need to be filled in individually, which is very time-consuming and prone to omissions. Therefore, we propose extracting elements (entities) from case transcripts using an entity classification model. This approach partially automates the process and reduces the time required for manual entry into the system. At the same time, it ensures that the elements (entities) are selected from the case transcript, reducing the possibility of typos during typing.

[0040] When using entity classification models to extract elements (entities) from case transcripts, a massive amount of case transcript text is required to construct a labeled sample set. The inventors discovered that a small number of elements (entities) have multiple entity types, meaning they have multiple labels, while a large number of entities have only one entity type, meaning they have a single label. Furthermore, only a small amount of manually annotated data can simultaneously label entities belonging to multiple entity types, creating a multi-label small sample problem. For example, a case transcript includes "suspect account number: 17712345678." Among them, "17712345678" can be the suspect's Alipay account name or the suspect's mobile phone number. Distinguishing the suspect's account type is essential. Therefore, an entity classification model is needed to map "17712345678" to multiple labels, such as "suspect's mobile phone number" and "suspect's Alipay account number." During the mapping process, determining whether it is a mobile phone number is easy, but determining whether it is an Alipay account number is difficult based on context. This results in the entity classification model predicting the mobile phone number as "suspect's mobile phone number" with a high probability, making it difficult to predict "suspect's Alipay account number." However, in real-world situations, when a mobile phone number is indeed an Alipay account number in the case transcript, predicting the "suspect's Alipay account" label is highly valuable. Therefore, the entity classification model needs to learn from small sample sizes to predict the entity and its multiple labels.

[0041] In order to solve the small sample problem of the above-mentioned multi-label classification, in some embodiments of this specification, a template text of the original text is constructed (used to illustrate an entity in the original text and multiple labels of the entity), and the number of outputs of the entity classification template is designed to adapt to the number of words in the template text (that is, one output corresponds to one word), and the entity classification model is trained with the goal of maximizing the probability score of each word in the template text, so that the entity classification model learns a small amount of information from the original text (including multi-label entities) to achieve multi-label classification of entities.

[0042] It should be noted that although the entities in the original text have multiple labels (for the convenience of description and distinction, the original text is called multi-label original text), the number of multi-label original texts is small, while the texts with single-label entities (for the convenience of description and distinction, the texts are called single-label original texts) are more. Based on this, the embodiment of this specification proposes the following technical ideas: train the entity classification model through a large amount of single-label original texts so that the entity classification model first learns the context information of the single label, and then train the entity classification model again through a small amount of multi-label original texts so that the entity classification model can fully learn the relationship between multiple labels based on the learned single-label context information, so that the entity classification model can make full use of the information of the small sample to learn the relationship between multiple labels, and more accurately predict the entity and the multiple labels of the entity, solving the problem of small samples in the above-mentioned multi-label classification. It is worth noting that through the above-mentioned technical ideas, the trained entity classification model can also predict the entity and the single label of the entity at the same time.

[0043] The following is a detailed description of the training scheme for the entity classification model provided in the embodiments of this specification.

[0044] Figure 2 FIG. 1 shows a schematic diagram of training an entity classification model in one embodiment. Figure 2 As shown in the figure, the process of training the entity classification model is as follows: construct a template text of the single-label original text (used to illustrate a single entity in the single-label original text and a label corresponding to the entity), use the single-label original text and its corresponding template text as a single-label training sample, and then obtain multiple single-label training samples. Figure 2 Single-label training samples 1, 2, ... are shown; a template text of a multi-label original text is constructed (used to illustrate a single entity in the original text and multiple labels corresponding to the entity), and the multi-label original text and its corresponding template texts are used as a multi-label training sample, thereby obtaining multiple multi-label training samples. Figure 2 Multi-label training samples 1, 2, ... are shown; two data sets are designed based on these training samples, one data set (for ease of distinction, called single-label sample set) includes only single-label training samples, and the other data set (for ease of distinction, called multi-label sample set) includes at least multi-label training samples; then, the entity classification model is trained based on the single-label sample set, so that the entity classification model learns the contextual information of the single label and obtains the trained entity classification model; thereafter, the trained entity classification model is trained based on the multi-label sample set, so that the entity classification model fully learns the relationship between multiple labels based on the learned single-label contextual information and obtains the final entity classification model.

[0045] It should be noted that the number of template texts corresponding to the above-mentioned single-label original text and multi-label original text is not specifically limited. The number of template texts needs to take into account the number of entities with labels in the text. Furthermore, in order to ensure that the entity classification model learns the difference between entities and non-entities, it may also be necessary to consider the number of non-entities (selected from content other than entities without labels in the text).

[0046] Here, the multi-label original text and the single-label original text are the original texts that require entity recognition and classification. The original texts can be manually entered or obtained by using OCR (Optical Character Recognition) technology to recognize images containing the original text. This embodiment of the present specification does not specifically limit this. For example, by performing OCR recognition on the above-mentioned case transcript file, multiple original texts can be obtained.

[0047] For example, the multiple tags for an entity in the multi-tag original text may include a mobile phone number and an application account. For example, the application may be Alipay, DingTalk, etc. It should be understood that the above multiple tags are merely examples and do not constitute a specific limitation. Multiple tags need to be determined based on actual scenarios.

[0048] Among them, the multi-label original text may also include single-label entities.

[0049] Template text can be a multi-tag template text, a single-tag template text, or a non-entity template text. Multi-tag template text is used to describe an entity and multiple tags for that entity. Single-tag template text is used to describe an entity and its single tag. Non-entity template text is used to describe non-entities other than entity types.

[0050] Specifically, a template library can be designed, and the multiple templates in the template library can be single-label templates (including one entity slot and one entity type slot), multi-label templates (including one entity slot and multiple entity type slots), and non-entity templates (including one entity slot); accordingly, for any entity with a label in the multi-label original text, based on the number of labels the entity has, a corresponding template is selected from the template library, and then the entity and the labels it has are automatically filled into the entity slot and entity type slot in the template to obtain the template text corresponding to the multi-label original text (for ease of distinction, called positive template text); then a positive template text is constructed for each entity with a label; further, for the multi-label original text, a non-entity template can also be selected, and the non-entities without labels in the multi-label original text (for example, consisting of several consecutive unlabeled words in the text, usually determined by random sampling) are filled into the entity slot in the non-entity template to obtain a template text (for ease of distinction, called negative template text). In practical applications, the number of negative template texts of the original text can be 1.5 times that of the positive template text. The method of obtaining the template text of a single-label original text is similar to that of a multi-label original text, and will not be repeated here.

[0051] For example, in a multi-label template, [candiate span] is both an [entity type] element and an [entity type] element; in a single-label template, [candiate span] is an [entity type] element; and in a non-entity template, [candiatespan] is not an element. [candidate span] represents an entity slot, and [entity type] represents an entity type slot.

[0052] Example 1: For a single-label original text: The bad guy Li called the victim Wang (mobile phone number 17733300000); among them, the label of [17733300000] is: [suspect's mobile phone number]; correspondingly, there is one positive template text corresponding to the original text, which is: 17733300000 is the suspect's mobile phone number; the corresponding negative template text can be multiple, such as: "A certain person called is not an element", "Giving the words to the victim is not an element", "Letting him give is not an element".

[0053] Example 2: For a multi-label original text: The victim transferred 5,000 yuan to Wang via Alipay 153*****00; the two labels of [153*****00] are [suspect's mobile phone number] and [suspect's Alipay account number]; correspondingly, there is one positive template text corresponding to the original text, which is: 153*****00 is both the suspect's mobile phone number and Alipay account number; there can be multiple corresponding negative template texts, for example, "Transfer to Wang is not an element", "Transfer is not an element", and "The victim's transfer to Wang is not an element".

[0054] Example 3, for multi-label original text (including entities with single labels): the bad guy Li called the victim Wang (mobile phone number 17733300000) and asked him to transfer 10,000 yuan to the Alipay account abc@163.com; among them, the label of [17733300000] is [suspect's mobile phone number], and the two labels of [abc@163.com] are [suspect's Alipay account] and [suspect's email address] respectively; correspondingly, there are two positive template texts corresponding to the original text, namely: 17733300000 is the suspect's mobile phone number; abc@163.com is both the suspect's Alipay account and the suspect's email address; the corresponding negative template texts can be multiple, for example: someone calling is not an element; words given to the victim are not an element; asking him to give is not an element.

[0055] It should be noted that this embodiment only takes two tags as an example and does not constitute a specific limitation. In actual applications, the number of tags for a single entity is not limited. It can be 3, 4, or even more, and the specific number needs to be determined based on actual conditions.

[0056] Among them, some or all training samples in the multi-label sample set are multi-label training samples. It should be noted that considering the small amount of multi-label original text, the model effect of the entity classification model may not be guaranteed. In actual applications, some training samples in the multi-label sample set are multi-label training samples, and some training samples are single-label training samples; for example, multi-label training samples account for 25% of the multi-label sample set, and single-label training samples account for 75% of the multi-label sample set.

[0057] It is worth noting that in order to ensure that the entity classification model can learn the relationship between different labels, preferably, the multiple labels of the entity in the multi-label original text include the single label of the entity in the single-label original text.

[0058] Among them, each training sample in the single-label sample set is a single-label training sample, and the single-label training sample here is a training sample outside the multi-label sample set. It is worth noting that all the labels of the single-label sample set can be completely different from all the labels of the multi-label sample set, or they can be partially the same, or they can be completely the same. For example, all the labels of the single-label sample set can include mobile phone number, email address, and Alipay account; all the labels of the multi-label sample set can include mobile phone number, email address, Alipay account, mobile phone number + email address, mobile phone number + Alipay account, and mobile phone number + email address + Alipay account.

[0059] Next, the training process of the entity classification model is described in detail.

[0060] First, the entity classification model is trained using a single-label sample set. In practical applications, for any single-label training sample in the single-label sample set, the single-label original text in the single-label training sample is used as the input of the entity classification model, and the entity classification model outputs the text sequentially to obtain the word probability distribution of each of the multiple moments (for the convenience of description and distinction, referred to as the first probability distribution); then, the order of the multiple moments is used as the order of the words in the template text in the single-label training sample to obtain the word probability distribution of each template text in the single-label training sample (for the convenience of description and distinction, referred to as the second probability distribution). The first probability distribution includes the probability scores of each of the preset multiple words; the second probability distribution includes the probability scores of each word in the corresponding template text.

[0061] Figure 3 FIG. 1 is a schematic diagram showing a probability distribution of a template text in one embodiment. Figure 3 As shown, the entity classification model is an end-to-end model, including an encoder and a decoder; assuming that the single-label training sample includes a single-label original text and N template texts; the single-label original text is input into the entity classification model, and the entity classification model sequentially outputs the first probability distribution P1, P2, ... at time t1, time t2, ..., to obtain the output result; based on the output result, the second probability distribution Q1, Q2, ..., Q N Here, the order of time t1, time t2, ... is adapted to the order of the characters in each template text. For example, time t1 corresponds to the first character c in template text 1. 11 , t2 time corresponds to the second word c in template text 1 12 The first probability distribution includes the probability scores of the preset multiple characters X1, X2, X3, ..., which are represented by vectors. The dimension of the vector is the same as the number of the preset multiple characters X1, X2, X3, .... The following describes how to determine the second probability distribution Q1 corresponding to the template text 1. The other template texts are similar and will not be described in detail. For the first character c in the template text 1 11, the word c in the first probability distribution P1 11 The probability score of word c in template text 1 11 The probability score Q 11 , and so on, determine the character c in template text 1 11 The letter c after 12 , ...their respective probability scores Q 12 , ..., and then obtain the second probability distribution Q1. It is worth noting that the number of outputs of the entity classification model, that is, the number of the first probability distribution, needs to meet the number of words in the template text with the largest number of words. For example, if the template text has 20 words, the entity classification template will have at least 20 outputs at the moment. In actual application, 20 first probability distributions will be output in sequence. It should be understood that in actual application, the vectorized representation of each word in the single-label original text is input into the encoder (not shown in the figure). The method of vectorized representation is described above. Figure 1 The description is not repeated here.

[0062] Then, for any template text, the template prediction loss of the template text is determined based on the second probability distribution of the template text; further, based on the template prediction loss corresponding to each template text in the single-label sample set, the model prediction loss is determined, and the entity classification model is trained with the goal of minimizing the model prediction loss, thereby maximizing the probability score of each word in each template text, and obtaining a trained entity classification model. Among them, the model prediction loss can be the average value of the template prediction loss of each template text in the single-label sample set. It should be noted that the trained entity classification model can learn the contextual information of a single label. In an example, the template prediction loss of any template text can be calculated specifically by the loss function shown in the following formula (1);

[0063]

[0064] Among them, P i，j represents the probability score of the jth word in the i-th template text; n represents the number of words in the i-th template text.

[0065] It should be noted that the above formula (1) is only an example of the loss function and does not constitute a specific limitation. The loss function can be reasonably designed in combination with the specific situation of the entity classification model.

[0066] The trained entity classification model is then further trained using the multi-label sample set to obtain the final entity classification model. The training process is similar to the training process for the entity classification model using the single-label sample set described above, differing only in the training samples, and will not be further described here. It should be noted that the final entity classification model is able to learn the connections between multiple labels and can predict a single entity and a single label or multiple labels for that entity.

[0067] In summary, the training method of the entity classification model provided in the embodiments of this specification can enable the entity type model to learn information of small samples with multiple labels and simultaneously predict multiple labels or a single label for a single entity.

[0068] Figure 4 A flow chart of a method for training an entity classification model according to an embodiment is shown, which can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. In order to facilitate the description of specific terms in the embodiments of this specification, such as training samples, original text, entities, entity types, sample sets, probability scores, prediction results, template texts, and entity classification models, first, second, ... are added before specific terms to indicate differences. Here, first, second, ... do not have special meanings and are only for the convenience of distinction and description. Figure 4 As shown, the method includes the following steps:

[0069] Step 41: Obtain a first sample set, which includes a first training sample. The first training sample includes a first original text and a first template text corresponding thereto. The first template text is used to describe a first entity in the first original text and a plurality of first entity types corresponding to the first entity.

[0070] The above-mentioned first sample set may correspond to the aforementioned multi-label sample set, which includes at least a first training sample (corresponding to the above-mentioned multi-label training sample). The first training sample includes a first original text (corresponding to the above-mentioned multi-label original text) and a corresponding first template text (corresponding to the above-mentioned positive template text), which is a text describing the first entity in the first original text and its corresponding multiple first entity types. In one example, the first template text can be obtained specifically by the following implementation method: determining the first entity in the first original text and the multiple first entity types corresponding to the first entity; then, obtaining a multi-type template adapted to the number of first entity types, the multi-type template including one entity slot and multiple entity type slots; thereafter, the first entity in the first original text and its corresponding multiple first entity types are filled into one entity slot and multiple entity type slots in the multi-type template to obtain a first template sample. For details, please refer to Example 2 and Example 3 above.

[0071] In some embodiments, the first training sample may further include a negative template text for the first original text, referred to as the second template text, which is a text describing non-entities other than the multiple first entity types in the first original text. This second template text may be generated using a corresponding template in a template library. Specifically, a non-entity template including an entity slot may be obtained; then, non-entities other than the first entity in the first original text may be filled into an entity slot in the non-entity template to obtain a second template sample. For details, please refer to the negative template text in Examples 1, 2, and 3 above.

[0072] In some possible embodiments, if the first original text also includes an entity of a single entity type, the first training sample also includes a corresponding positive template text, which is a text representing the entity in the first original text and the single entity type corresponding to the entity. In this way, the first training sample may include multiple template texts.

[0073] Next, in step 42, the first original text is input into a first entity classification model to obtain a first prediction result, which includes sequentially outputting a first probability score of each word in the first template text.

[0074] According to a feasible implementation, the first entity classification model includes an encoder and a decoder, wherein the encoder is used to encode the input text to obtain an encoding vector, and the decoder is used to decode the encoding vector to output the word probability distribution corresponding to each of multiple moments (corresponding to the above-mentioned first probability distribution).

[0075] Correspondingly, when the first original text is input into the above-mentioned first entity classification model, the encoder encodes the first original text to obtain an encoding vector; the decoder decodes the encoding vector and outputs the word probability distribution corresponding to each of the multiple moments. The first probability score can be determined based on the word probability distribution corresponding to each of the multiple moments. Specifically, the order of the aforementioned multiple moments is made to correspond to the order of each word in the first template text. For each target moment in the multiple moments, the probability of the target word is determined from the word probability distribution corresponding to the target moment, and included in the first probability score. The target word is the word in the first template text whose sequence number corresponds to the target moment. In this way, the first probability score of each word in the first template text output by the first entity classification model in sequence can be included in the above-mentioned first prediction result.

[0076] When the first training sample also includes the aforementioned second template text (negative template text), a second probability score for each character in the second template text can be determined based on the probability distribution of the characters corresponding to the multiple time instants output by the decoder, and included in the first prediction result for the first training sample. The second probability score is determined similarly to the first probability score, except that the target character at any target time instant is the character in the second template text with the sequence number corresponding to the target time instant.

[0077] When the first training sample also includes other template texts, a similar method can be used to determine the probability score for the template text and include it in the first prediction result.

[0078] Then, in step 43, a first prediction loss corresponding to the first training sample is determined based on the first prediction result. The prediction loss can be determined in the form of various loss functions, which are not limited here.

[0079] The above describes in detail the process of determining the prediction loss for a single training sample (multi-label sample). For each sample in the first sample set, a prediction loss can be determined. Then, in step 44, the first entity classification model is trained based on the prediction loss corresponding to each training sample in the first sample set.

[0080] Specifically, the model prediction loss can be determined based on the prediction loss corresponding to each training sample in the first sample set, for example, the sum of the prediction losses corresponding to each training sample in the first sample set; then, the entity classification model is trained with the goal of minimizing the model prediction loss, thereby maximizing the probability score of each word in each template text, and obtaining a trained entity classification model.

[0081] The above describes the processing process for multi-label samples in a multi-label sample set. As mentioned above, in some embodiments, the first sample set may also include a second training sample (corresponding to the above-mentioned single-label training sample). The second training sample includes a second original text (corresponding to the above-mentioned single-label original text), and a third template text corresponding to the second original text (corresponding to the above-mentioned positive template text), which is a text describing the second entity in the second original text and its corresponding single second entity type. In one example, the third template text can be determined specifically by the following implementation method: obtain a single-type template including an entity slot and an entity type slot; then, fill the second entity in the second original text and its corresponding single second entity type into an entity slot and an entity type slot in the single-type template to obtain a third template sample. For specific examples, please refer to Examples 1 and 3 above.

[0082] In some examples, the second training sample may also include a fourth template text (corresponding to the negative template text described above), which is text describing non-entities other than the second entity type in the second original text. The method for obtaining this fourth template text is similar to the process for obtaining the second template text described above and will not be repeated here.

[0083] When the first sample set includes the above-mentioned second training sample, the model training process also includes: inputting the second original text into the above-mentioned first entity classification model to obtain a second prediction result, which includes sequentially outputting the third probability score of each word in the third template text.

[0084] Similar to the above-mentioned method of obtaining the first prediction result, when the second original text is input into the first entity classification model, the decoder outputs the word probability distribution corresponding to each of the multiple moments for the second original text, and the second prediction result can be determined based on the word probability distribution. Specifically, each word in the third template text can be used as the target word at each moment, and the third probability score of each word in the third template text can be determined to be output in sequence based on the word probability distribution corresponding to each of the above-mentioned multiple moments, and included in the above-mentioned second prediction result. When the second training sample also includes a fourth template text, each word in the fourth template text can be used as the target word at each moment, and the fourth probability score of each word in the fourth template text can be determined to be output in sequence, and included in the second prediction result.

[0085] In this way, the second prediction loss corresponding to the second training sample can be determined based on the above second prediction result.

[0086] In this case, the prediction loss corresponding to each training sample in the first sample set in step 44 includes not only the first prediction loss corresponding to each first training sample, but also the second prediction loss corresponding to each second training sample. Thus, the first entity classification model is trained based on the combined sample set containing multi-label samples and single-label samples.

[0087] According to a feasible implementation, the first entity classification model is a model pre-trained using a large number of single-label samples. In this implementation, before the aforementioned step 41, the following process is also performed.

[0088] A second sample set (corresponding to the above-mentioned single-label sample set) is obtained, in which each third training sample included therein (corresponding to the single-label training sample in the above-mentioned single-label sample set) includes a third original text and a corresponding fifth template text (corresponding to the above-mentioned positive template text), which is a text describing the third entity in the corresponding third original text and the single entity type corresponding to the third entity; its acquisition method is similar to the aforementioned process of obtaining the third template text; in some examples, the third training sample may also include a sixth template text (corresponding to the above-mentioned negative template text), which is a text describing non-entities other than the third entity type in the third original text. The acquisition method is similar to the aforementioned process of obtaining the second template text, and will not be repeated here; then, for each third original text of the second sample set, the third original text is input into the second entity classification model (initial model) to obtain a third prediction result, and the acquisition method is similar to the aforementioned first prediction result; wherein, the result at least includes sequentially outputting the fifth probability score of each word of the fifth template text; if the third original text also corresponds to the sixth template text; then the third prediction result may also include sequentially outputting the sixth probability score of each word of the sixth template text; thereafter, based on the third prediction results corresponding to each third training sample in the second sample set, the second entity classification model is trained, and the trained second entity classification model is used as the first entity classification model. The first entity classification model corresponds to the entity classification model trained based on the single-label sample set mentioned above. Here, the model structure of the first entity classification model and the second entity classification model are the same, but the model parameters are different.

[0089] Looking back at the above process, in the embodiments of this specification, a template text of the original text is constructed (used to illustrate an entity in the original text and multiple labels of the entity), and the number of outputs of the entity classification template is designed to adapt to the number of words in the template text (that is, one output corresponds to one word), and the entity classification model is trained with the goal of maximizing the probability score of each word in the template text, so that the entity classification model learns a small amount of information from the original text (including multi-label entities) to realize multi-label classification of entities.

[0090] According to another embodiment, a device for training an entity classification model is also provided. Figure 5 The schematic diagram of the structure of the entity classification model training device according to one embodiment is shown. The device can be deployed in any device, platform or device cluster with data storage, computing and processing capabilities. Figure 5 As shown, the device 500 includes:

[0091] The sample set acquisition module 51 is configured to acquire a first sample set, including a first training sample, wherein the first training sample includes a first original text and a first template text corresponding thereto, wherein the first template text is used to describe a first entity in the first original text and a plurality of first entity types corresponding to the first entity;

[0092] A prediction module 52 is configured to input the first original text into a first entity classification model to obtain a first prediction result, including sequentially outputting a first probability score of each word in the first template text;

[0093] a loss determination module 53 configured to determine a first prediction loss corresponding to the first training sample based on the first prediction result;

[0094] The training module 54 is configured to train the first entity classification model based on the prediction loss corresponding to each training sample in the first sample set.

[0095] In various embodiments, the above modules are specifically configured to perform the above combined Figure 3 The various steps in the described method will not be repeated here.

[0096] Through the above device, a multi-label template text of the original text is constructed (used to describe an entity in the original text and multiple labels of the entity), and the constructed multi-label template text is used as the output of the entity classification model to predict the probability score of each word in the multi-label template text. The entity classification model is trained with the goal of maximizing the probability score of each word in the multi-label template text, so that the entity classification model learns a small amount of information from the original text (including multi-label entities) to realize multi-label classification of entities.

[0097] According to another embodiment, there is also provided a computer readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute a combination of Figure 3 The method described.

[0098] According to another embodiment, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the system realizes the combination of Figure 3 The method described.

[0099] Those skilled in the art will appreciate that, in one or more of the above examples, the functions described herein may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0100] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for training an entity classification model, comprising: Acquire a first sample set, including a first training sample, wherein the first training sample includes a first original text and a first template text corresponding thereto, wherein the first template text is used to describe a first entity in the first original text and a plurality of first entity types corresponding to the first entity; Inputting the first original text into a first entity classification model to obtain a first prediction result, which includes sequentially outputting a first probability score of each word in the first template text; Determining a first prediction loss corresponding to the first training sample based at least on the first prediction result; The first entity classification model is trained based on the prediction loss corresponding to each training sample in the first sample set.

2. The method according to claim 1, wherein The first training sample further includes a second template text, where the second template text is used to describe non-entities in the first original text that do not belong to the multiple first entity types; The first prediction result also includes sequentially outputting a second probability score of each word in the second template text.

3. The method according to claim 1, wherein The first sample set further includes a second training sample, the second training sample including a second original text and a corresponding third template text, the third template text being used to describe a second entity in the second original text and a single second entity type corresponding to the second entity; The method further includes: inputting the second original text into the first entity classification model to obtain a second prediction result, which includes sequentially outputting a third probability score of each word in the third template text; A second prediction loss corresponding to the second training sample is determined based at least on the second prediction result.

4. The method according to claim 3, wherein: The second training sample further includes a fourth template text, and the fourth template text is used to describe non-entities other than the second entity type in the second original text; The second prediction result also includes sequentially outputting a fourth probability score of each word in the fourth template text.

5. The method according to claim 3, wherein: The plurality of first entity types includes the second entity type.

6. The method according to claim 3, wherein: The method further comprises: Obtain a single-type template, the single-type template including an entity slot and an entity type slot; The second entity in the second original text and its corresponding single first entity type are respectively filled into the entity slot and entity type slot in the single-type template to obtain the third template text.

7. The method according to claim 1, wherein The method further comprises: Obtaining a multi-type template corresponding to the first original text, the multi-type template including an entity slot and multiple entity type slots; The first entity in the first original text and its corresponding multiple first entity types are respectively filled into the entity slot and multiple entity type slots in the multi-type template to obtain the first template sample; the number of the multiple entity type slots is the same as the number of the multiple first entity types.

8. The method according to claim 1, wherein The method further comprises: Obtaining a second sample set, which includes a plurality of third training samples, each third training sample includes a third original text and a corresponding fifth template text, wherein the fifth template text is used to describe a third entity in the corresponding third original text and a single third entity type corresponding to the third entity; Inputting the third original text into a second entity classification model to obtain a third prediction result, which includes sequentially outputting a fifth probability score of each word in a fifth template text; training the second entity classification model based on the third prediction results corresponding to each third training sample in the second sample set, and using the trained second entity classification model as the first entity classification model; The training of the first entity classification model based on the prediction loss corresponding to each training sample in the first sample set includes: The first entity classification model is trained based on the prediction loss corresponding to each training sample in the first sample set to obtain a trained first entity classification model.

9. The method according to claim 8, wherein The third entity type is any type of the multiple first entity types; The number of samples in the second sample set is greater than the number of samples in the first sample set.

10. The method according to claim 1, wherein The first entity classification model includes an encoder and a decoder; the first original text is input into the first entity classification model to obtain a first prediction result, including: Encoding the first original text using the encoder to obtain an encoding vector; Decoding the encoding vector using the decoder to output word probability distributions corresponding to multiple moments; For each target moment among the multiple moments, a probability of a target word is determined from the word probability distribution corresponding to the target moment and included in the first probability score, wherein the target word is a word whose sequence number in the first template text corresponds to the target moment.

11. The method according to claim 1, wherein The multiple first entity types include mobile phone numbers and application accounts.

12. A training device for an entity classification model, comprising: a sample set acquisition module configured to acquire a first sample set, including a first training sample, wherein the first training sample includes a first original text and a first template text corresponding thereto, wherein the first template text is used to describe a first entity in the first original text and a plurality of first entity types corresponding to the first entity; a prediction module configured to input the first original text into a first entity classification model to obtain a first prediction result, including sequentially outputting a first probability score of each word in the first template text; a loss determination module, configured to determine a first prediction loss corresponding to the first training sample based at least on the first prediction result; The training module is configured to train the first entity classification model based on the prediction loss corresponding to each training sample in the first sample set.

13. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 11.

14. A computing device comprising a memory and a processor, characterized in that: The memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 11 is implemented.

Images