Named entity recognition method, apparatus, device, medium and program product

Abstract

The application relates to the technical field of artificial intelligence, and discloses a named entity recognition method, device, equipment, medium and program product, which are used for performing a named entity recognition task based on a model of distillation learning. The method part comprises the following steps: obtaining an identified sentence; inputting the identified sentence into a target neural network model which has been trained to obtain first prediction probability values of each first annotation label of a first label set to which a word of the identified sentence belongs and second prediction probability values of each second annotation label of a second label set; taking a label type of a first annotation label with the largest first prediction probability value as a first prediction label type of the word, and taking a label type of a second annotation label with the largest second prediction probability value as a second prediction label type of the word; and determining a final prediction label type of the word according to the first prediction label type and the second prediction label type of the word.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a named entity recognition method, apparatus, device, medium, and program product. Background Technology

[0002] Named entity recognition (NER) has a wide range of applications. NER involves identifying named entities of interest within given text, such as place names, personal names, company names, times, and amounts. Traditional NER methods utilize a pre-trained Bidirectional Encoder Representation from Transformers (BERT) network combined with a Long Short-Term Memory (LSTM) network. LSTM is a special type of recurrent neural network (RNN).

[0003] In the traditional approach described above, when the application scenario changes and a named entity recognition model is used, training the old model on new data will generally reduce its ability to predict old label types. Moreover, adding new parameters to the old model to predict new label types requires re-labeling a large amount of all newly introduced data in the context of the integrated large dataset and larger set of label types. This consumes a lot of additional data labeling and processing resources, and in the early stages of achieving the named entity recognition task, too much data labeling and processing resources are required. Summary of the Invention

[0004] This application relates to the field of artificial intelligence technology and provides a named entity recognition method, apparatus, device, medium, and program product to solve the technical problem that traditional solutions require re-annotating a large amount of all newly introduced data, which occupies a large amount of additional data annotation and processing resources.

[0005] A named entity recognition method, the method comprising:

[0006] Obtain the recognized sentence;

[0007] The sentence to be identified is input into the trained target neural network model to obtain the first predicted probability value of each first label of the first label set and the second predicted probability value of each second label of the second label set.

[0008] The label type of the first label with the largest first prediction probability value is taken as the first predicted label type of the word, and the label type of the second label with the largest second prediction probability value is taken as the second predicted label type of the word.

[0009] The final predicted label type of a word is determined based on its first and second predicted label types.

[0010] The target neural network model is obtained by distilling the student network model using a pre-trained named entity recognition model as the teacher network model.

[0011] Optionally, both the first and second label sets use BIO sequence labeling. Based on the first and second predicted label types of the words, the final predicted label type of the words is determined, including:

[0012] If both the first and second predicted label types are label type O, then the final predicted label type of the word is determined to be label type O.

[0013] If one of the first predicted label type and the second predicted label type is label type B and the other is label type O, then the final predicted label type of the word is determined to be label type B.

[0014] When both the first predicted label type and the second predicted label type are label type B, the label type with the highest predicted probability value is selected as the final predicted label type from the first predicted label type and the second predicted label type.

[0015] If either the first predicted label type or the second predicted label type is label type I, then the final predicted label type is determined based on the label type of the preceding word and the entity naming category.

[0016] Optionally, the final predicted label type is determined based on the label type of the preceding word and the entity naming category, including:

[0017] If the label type of the preceding word is label type B or label type I, and the entity naming category of the word is the same as the entity naming category of the preceding word, then the final predicted label type is determined to be label type I.

[0018] If the label type of the preceding word is label type O, or if the naming category of the word is different from the entity naming category of the preceding word, then one of the predicted label types is changed to label type O, and a label prediction is performed with the other predicted label type among the first and second predicted label types to determine the final predicted label type.

[0019] Optionally, the target neural network model is trained in the following manner:

[0020] The training sentences are input into the pre-trained teacher network model to obtain the third predicted probability value of each word in the training sentences belonging to the first labeled label in the first label set;

[0021] The training sentences are input into the student network model to obtain the fourth prediction probability value of each word in the training sentences belonging to the first label and the fifth prediction probability value of each word belonging to the second label of the second label set.

[0022] The first loss value is calculated based on the fifth predicted probability value of each word and the true value of each word belonging to the second label, and the second loss value is calculated based on the third predicted probability value of each word belonging to the first label and the fourth predicted probability value of each word belonging to the first label.

[0023] The training of the student network model is constrained by the first loss value and the second loss value until the student network model meets the preset conditions. The student network model that meets the preset conditions is then used as the target neural network model.

[0024] Optionally, the named entity categories corresponding to the entity annotation tags in the first tag set are different from the named entity categories corresponding to the entity annotation tags in the second tag set.

[0025] Optionally, the training of the student network model is constrained according to the first loss value and the second loss value until the student network model meets preset conditions, including:

[0026] Obtain the first weight corresponding to the first loss value, and obtain the second weight corresponding to the second loss value;

[0027] Calculate the total loss value based on the first loss value, the second loss value, the first weight, and the second weight;

[0028] The training of the student network model is constrained by the total loss value until the student network model meets the preset conditions.

[0029] Optionally, the total loss value is calculated based on the first loss value, the second loss value, the first weight, and the second weight, including:

[0030] The total loss value is obtained using the following formula:

[0031] L=αL1(y,p 学生,E’ )+βL2(p 学生,E ,p 老师,T,E );

[0032] L represents the total loss, α represents the first weight, β represents the second weight, y represents the true value, and p 学生,E’ p represents the fifth predicted probability value belonging to the second label. 学生,E p represents the third predicted probability value belonging to the first labeled item. 老师,T,E L1(y,p) represents the fourth predicted probability value belonging to the first labeled element. 学生,E’ L2(p) represents the first loss value.学生,E ,p 老师,T,E ) represents the second loss value, and T represents the distillation temperature parameter that constrains the output smoothness of the fourth prediction probability value.

[0033] Optionally, the teacher network model includes a pre-trained first feature extraction network model, a first linear layer, and a first normalization layer. The output of the first feature extraction network model is connected to the input of the first linear layer, and the output of the first linear layer is connected to the input of the first normalization layer. The student network model includes a second feature extraction network model, a second linear layer, a second normalization layer, a third linear layer, and a third normalization layer. The output of the second feature extraction network model is connected to the inputs of the second linear layer and the third linear layer, respectively. The output of the second linear layer is connected to the input of the second normalization layer, and the output of the third linear layer is connected to the input of the third normalization layer.

[0034] Optionally, the initial training parameters of the second feature extraction network model are derived from the first feature extraction network model, and the initial training parameters of the second linear layer are derived from the first linear layer.

[0035] A named entity recognition device, comprising:

[0036] The acquisition module retrieves the sentence to be recognized.

[0037] The input module is used to input the sentence to be recognized into the trained target neural network model to obtain the first predicted probability value of each first label of the words in the sentence to be recognized belonging to the first label set, and the second predicted probability value of each second label of the words in the second label set.

[0038] The filtering module is used to take the label type of the first label with the largest first prediction probability value as the first predicted label type of the word, and take the label type of the second label with the largest second prediction probability value as the second predicted label type of the word.

[0039] The determination module is used to determine the final predicted label type of a word based on the first predicted label type and the second predicted label type of the word.

[0040] The target neural network model is obtained by distilling the student network model using a pre-trained named entity recognition model as the teacher network model.

[0041] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the aforementioned training method or named entity method.

[0042] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the aforementioned training method or named entity method.

[0043] A computer program product comprising a computer program, characterized in that, when executed by a processor, the computer program implements the steps of the aforementioned training method or named entity method.

[0044] This application provides a novel named entity recognition method. By using the trained student network model as the target neural network model, the two predicted label types output by the student network model are combined for prediction and output. This method makes good use of the predictive ability of the teacher network model. It can obtain the required student network model for label prediction without spending additional labeling time, effort and training time. It does not require too much data labeling work during training. Moreover, the student network model can also be used for the prediction output of new labels to achieve the named entity recognition task. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a flowchart illustrating a named entity recognition method according to an embodiment of this application;

[0047] Figure 2 This is a schematic diagram of a training process for a student network model in one embodiment of this application;

[0048] Figure 3 This is a schematic diagram of a training framework for a student network model in one embodiment of this application;

[0049] Figure 4 This is a schematic diagram of a named entity recognition method apparatus according to an embodiment of this application;

[0050] Figure 5 This is a schematic diagram of a computer device according to one embodiment of this application. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] This embodiment provides a named entity recognition method and related devices and products, which are described below through specific implementation methods.

[0053] In one embodiment, such as Figure 1 As shown, a named entity recognition method is provided, which specifically includes the following steps:

[0054] S101: Obtain the sentence to be recognized.

[0055] S102: Input the sentence to be identified into the trained target neural network model to obtain the first predicted probability value of each first label of the words in the sentence to be identified belonging to the first label set, and the second predicted probability value of each second label of the words to be identified belonging to the second label set.

[0056] The target neural network model is trained by inputting the sentence to be recognized into it, and the first predicted probability value of each first label (E) belonging to the first label set and the second predicted probability value of each second label (E') belonging to the second label set are obtained. The target neural network model is obtained by distilling a pre-trained named entity recognition model as the teacher network model onto the student network model. In this embodiment, the teacher network model is denoted as M. 老师 Let M be the student network model. 学生 .

[0057] S103: Take the label type of the first label with the largest first prediction probability value as the first predicted label type of the word, and take the label type of the second label with the largest second prediction probability value as the second predicted label type of the word.

[0058] S104: Determine the final predicted label type of a word based on its first and second predicted label types.

[0059] In this embodiment, after training the student network model M... 学生 Then, the student network model M can be used. 学生 As a target neural network model used in named entity recognition, the sentence to be recognized is first input into the target neural network model (i.e., the student network model M). 学生The first predicted probability value of the words in the identified sentence belonging to each first label of the first label set E, and the second predicted probability value of each second label of the second label set E' are obtained.

[0060] In one embodiment, the named entity categories corresponding to the entity annotation tags in the first tag set E are different from the named entity categories corresponding to the entity annotation tags in the second tag set E'. For example, assuming the BIO tagging method is used for named entity recognition, where BX represents the beginning of named entity X, IX represents the end of entity X, and O represents not belonging to any type. For instance, if X represents a noun phrase (such as a place name), then the three BIO tags are: B-NP: the beginning of the noun phrase (place name), I-NP: the middle of the noun phrase (place name), and O: not a noun phrase (place name). In this embodiment, for example, the first tag set E (e.g., includes O, B-company name, I-company name), and the second tag set E' (e.g., includes O, B-place name, I-place name). It should be noted that the above example is merely illustrative; in actual applications, the tag types of the annotation tags in the first tag set and the annotation tags in the second tag set can also be different. For example, the first tag set E (e.g., including O, B-animal name, I-animal name), the second tag set E' (e.g., including O, B-place name, I-place name), and so on, without any specific limitations.

[0061] The first tag set E and the second tag set E' each include their own annotation tags. To distinguish and clearly describe this embodiment, the annotation tags in the first tag set are referred to as the first annotation tags, and the annotation tags in the second tag set are referred to as the second annotation tags. The annotation tags in the two tag sets generally include multiple tags, depending on the tag annotation method used. In addition, it should be noted that words can refer to a single character, a phrase composed of a single character or multiple characters, a string composed of multiple characters, etc., and this embodiment of the application is not limited to this.

[0062] The target neural network model is configured to identify the predicted probability value of each word in the identified sentence belonging to the first label in the first label set E, denoted as the first predicted probability value, and to identify the predicted probability value of each word in the identified sentence belonging to the second label in the second label set E', denoted as the second predicted probability value.

[0063] To facilitate understanding of the student network model M 学生 The training process will be described below:

[0064] In one embodiment, such as Figure 2 As shown, a training method for a named entity recognition model is provided, which specifically includes the following steps:

[0065] S10: Input the training sentence into the pre-trained teacher network model to obtain the third predicted probability value of each word in the training sentence belonging to the first label in the first label set.

[0066] S20: Input the training sentence into the student network model to obtain the fourth prediction probability value of each word in the training sentence belonging to the first label and the fifth prediction probability value of each word belonging to the second label of the second label set.

[0067] S30: Calculate the first loss value based on the fifth predicted probability value of the second label and the true value of the second label, and calculate the second loss value based on the third predicted probability value and the fourth predicted probability value of the first label.

[0068] S40: The training of the student network model is constrained according to the first loss value and the second loss value until the student network model meets the preset conditions. The student network model that meets the preset conditions is taken as the target neural network model.

[0069] To facilitate understanding of the above steps, the training network framework used in the embodiments of this application needs to be introduced here. One difference between the embodiments of this application and the conventional solutions described in the background art is that the network framework of the embodiments of this application sets two models: one is a pre-trained old model used to guide the training of the new model, denoted as the teacher network model M. 老师 The other is the newly trained model, denoted as the student network model M. 学生 Among them, the teacher network model M 老师 The predicted probability value is configured to identify the first label in the first label set E for each word in the input sentence, and is denoted as the third predicted probability value; while the student network model M 学生 The predicted probability value configured to identify each word in the input sentence belonging to the first label in the first label set is denoted as the fourth predicted probability value, and the predicted probability value to identify each word in the input sentence belonging to the second label in the second label set E' is denoted as the fifth predicted probability value.

[0070] During training, input sentences are needed as training data, denoted as training sentences, and then the training sentences are input into the pre-trained teacher network model M. 学生 In the process, the third predicted probability value of each word in the training sentence belonging to the first label in the first label set E is obtained, and the same training sentence is then input into the student network model M. 学生In the training sentence, the fourth predicted probability value for each word belonging to the first label and the fifth predicted probability value for each word belonging to the second label set E' are obtained. Then, based on the fifth predicted probability value of the second label and the true value of the second label, the first loss value is calculated, and based on the third and fourth predicted probability values ​​of the first label, the second loss value is calculated. In other words, the first loss value represents the difference between the output probability value and the true value for the same word belonging to the second label, and the second loss value represents the difference between the output probability value and the true value for the same word belonging to the first label in the student network model M. 学生 With teacher network model M 老师 The loss between the output probability values ​​is used to constrain the student network model M based on the first and second loss values. 学生 Training continues until the student network model M is reached. 学生 If the preset conditions are met, the student network model M will meet the preset conditions. 学生 As the target neural network model.

[0071] As can be seen, this embodiment provides a novel training method for a named entity recognition model, which utilizes a pre-trained teacher network model M. 老师 To guide the new student network model M 学生 The training will not reduce the performance of the old teacher network model M. 老师 The ability to predict old labels leads to forgetting problems. Moreover, the above training process does not require additional work on re-labeling the training data, which greatly reduces processing resources and also helps to reduce data labeling time and training time.

[0072] In one embodiment, such as Figure 3 As shown in the embodiments of this application, a specific network framework is provided, namely the teacher network model M. 老师 This includes a pre-trained first feature extraction network model N1, a first linear layer L1, and a first normalization layer S1. The output of the first feature extraction network model N1 is connected to the input of the first linear layer L1, and the output of the first linear layer L1 is connected to the input of the first normalization layer S1. A student network model M... 学生It includes a second feature extraction network model N2, a second linear layer L2, a second normalization layer S2, a third linear layer L3, and a third normalization layer S3. In this system, the first linear layer L1 outputs the predicted score (e.g., logit value) for each word in the training sentence belonging to the first label in the first label set E. The first normalization layer S1 outputs the corresponding third predicted probability value based on the predicted score output by the first linear layer L1. The output of the second feature extraction network model N2 is connected to the inputs of the second linear layer L2 and the third linear layer L3, respectively. The output of the second linear layer L2 is connected to the input of the second normalization layer S2, and the third linear layer L3 is connected to the input of the third normalization layer S3. The second linear layer L2 outputs the predicted score (e.g., logit value) for each word in the training sentence belonging to the first label. The second normalization layer S2 outputs the corresponding fourth predicted probability value based on the predicted score output by the second linear layer L2. The third linear layer L3 outputs the predicted score (e.g., logit value) for each word in the training sentence belonging to the second label in the second label set E'. The third normalization layer S3 outputs the corresponding fifth predicted probability value based on the predicted score output by the third linear layer L2.

[0073] It should be noted that, based on the above embodiments, in practical applications, the teacher network model M 老师 And student network model M 学生 There can be multiple implementations, meaning the first feature extraction network model N1 and the second feature extraction network model N2 can have multiple implementations, and the corresponding linear layers and normalization layers can also have multiple implementations; no specific limitation is made. In one embodiment, the first feature extraction network model N1 can use a pre-trained BERT model, the first linear layer L1 can be a fully connected layer, and the first normalization layer S1 can be a softmax layer, using the softmax layer to output the third predicted probability value; similarly, the second feature extraction network model N2 also uses a BERT model, the second linear layer L2 can be a fully connected layer, and the second normalization layer S2 and the third normalization layer S3 can each be a softmax layer, using the student network model M 学生 These two softmax layers output the fourth and fifth predicted probability values, respectively.

[0074] It is understandable that the core of the BERT model is a Transformer language model that includes a variable number of encoder layers and self-attention heads. It should be noted that in other embodiments, the first feature extraction network model N1 and / or the first feature extraction network model N2 can also be other models, such as the GTP model (Generative Pre-Training), etc., without limitation. Furthermore, the implementation of each linear layer and the unified layer can also have corresponding variations, which are not specifically limited here.

[0075] In one embodiment, the initial training parameters of the second feature extraction network model N2 are derived from the first feature extraction network model N1, and the initial training parameters of the second linear layer L2 are derived from the first linear layer L1. Since training from scratch is time-consuming and unnecessary, training typically begins with a pre-trained model, while the teacher network model M... 老师 It provides a good reference, so the teacher's network model M can be used directly during training. 老师 The relevant parameters are used as the student network model M 学生 The initial training parameters are beneficial for further improving training speed.

[0076] It should also be noted that Figure 2 The network structure described here is merely illustrative and does not constitute a limitation. For example, the first normalization layer S1, the second normalization layer S2, and the third normalization layer S3 can represent the same normalization layer independent of the teacher network model M. 老师 And student network model M 学生 It exists, but is only used to calculate the corresponding softmax value (i.e., the predicted probability value) based on the logit value at the end, and is not specifically limited.

[0077] As can be seen from the above embodiments, the embodiments of this application provide a specific teacher network model M. 老师 And student network model M 学生 The network structure makes the solution feasible.

[0078] In one embodiment, during training, the named entity categories corresponding to the entity annotation labels in the first label set are different from those corresponding to the named entity categories in the second label set. For example, suppose BIO tags are used for named entity recognition, where BX represents the beginning of named entity X, IX represents the end of entity X, and O represents not belonging to any type. For instance, if X represents a noun phrase (such as a place name), then the three BIO tags are: B-NP: the beginning of the noun phrase (place name), I-NP: the middle of the noun phrase (place name), and O: not a noun phrase (place name). In this embodiment, for example, the first label set E (e.g., including O, B-company name, I-company name) and the second label set E' (e.g., including O, B-place name, I-place name). It should be noted that the above example is merely illustrative; in actual applications, the label types of the annotation labels in the first label set and the annotation labels in the second label set can also be different in other ways. For example, the first tag set E (e.g., including O, B-animal name, I-animal name), the second tag set E' (e.g., including O, B-place name, I-place name), and so on, without any specific limitations.

[0079] It is also important to emphasize that this embodiment can solve the problem of identifying new named entity categories based on new input data on an already trained model. For example, it can train the ability to identify company names on the basis of a model that has already been trained to recognize people's names, and ensure that the prediction ability of the newly trained model does not decrease compared to the original model that recognizes people's names. In traditional solutions, it is often necessary to retrain the model and require a considerable amount of data of new categories to achieve the goal. In addition, continuing to train on a new dataset may lead to catastrophic forgetting problems, because the new data contains fewer old entity types, which will affect the ability to recognize old entity types.

[0080] It should be noted that the training sentences are input into the pre-trained teacher network model M. 老师 In the process, the third predicted probability value of each word in the training sentence belonging to the first labeled label in the first label set E is obtained, depending on the network architecture used, based on... Figure 3 The network architecture shown generally includes the following processes: Teacher network model M 老师 The BERT model obtains the feature output of each word in the input sentence (with dimensions n*d, where n represents the number of characters and d represents the embedding dimension). A linear layer then connects these features to obtain the predicted score (logit value) for each character, and a softmax layer completes the process to obtain an n*1 vector. Teacher's network model M 老师 The output third predicted probability value can be expressed as p 老师,T,E The first predicted probability of the j-th element of the n*1 vector is represented as p. j,老师,T,E =exp(z j / T) / ∑ k exp(z k / T), z j The predicted score (logit value) for a word belonging to label type j is z. k This represents the sum of predicted scores (logit values) for words belonging to all label types. T represents the distillation temperature parameter. A higher T results in a smoother output probability distribution of the softmax function, with a greater entropy. This value can be set according to requirements and is not limited here. A larger zi value for a certain category indicates a higher probability of belonging to that category. The summation score zj for each category represents the probability of belonging to the j-th category. Because logit values ​​are not probability values, the softmax function is generally used to transform the logit values, and the resulting predicted probability value is used as the final probability, i.e., the third predicted probability value. It should also be noted that the student network model M... 学生The calculation methods for the corresponding fourth and fifth prediction probability values ​​are the same as or similar to those for the third prediction probability value, and will not be repeated here.

[0081] Understandably, large models are generally single complex networks or collections of networks, possessing excellent performance and generalization ability, while small models, due to their smaller network size, have limited expressive power. Therefore, the knowledge learned by large models can be used to guide the training of smaller models, as in the embodiment of this application using the teacher network model M. 老师 To guide and train student network model M 学生 This allows the student network model to achieve performance comparable to a larger model, but with a significantly reduced number of parameters, thus achieving model compression and acceleration to obtain the desired target neural network model. In this embodiment, the distillation temperature parameter T changes the student network model M. 学生 The degree of attention paid to specific labels during training, in general, depends on the choice of the distillation temperature parameter T and the student network model M. 学生 The size is related to the student network model M 学生 When the number of parameters is relatively small, a relatively low distillation temperature parameter T is sufficient. This is because a model with a small number of parameters cannot learn from all the teacher network models M. 老师 The knowledge can be obtained by setting the distillation temperature parameter T (e.g., lowering it) to reasonably train the student network model M. 学生 .

[0082] In one embodiment, step 40, namely, constraining the training of the student network model according to the first loss value and the second loss value until the student network model meets the preset conditions, specifically includes the following steps:

[0083] S41: Obtain the first weight corresponding to the first loss value, and obtain the second weight corresponding to the second loss value.

[0084] S42: Calculate the total loss value based on the first loss value, the second loss value, the first weight, and the second weight.

[0085] S43: Use the total loss value to constrain the training of the student network model until the student network model meets the preset conditions.

[0086] In this embodiment, two hyperparameters are set for the training process: a first weight α and a second weight β. These two weights are used to control the first and second loss values, respectively. These weights are empirical values ​​and are not specifically limited here. After setting these two weight values, the total loss value is calculated based on the first loss value, the second loss value, the first weight, and the second weight. It should be noted that in some embodiments, weights may not be set, and the total loss value may be calculated directly based on the first and second loss values; this is not specifically limited.

[0087] Specifically, in one embodiment, step S41, namely calculating the total loss value based on the first loss value, the second loss value, the first weight, and the second weight, includes obtaining the total loss value using the following formula: L=αL1(y,p 学生 ,E')+βL2(p 学生,E ,p 老师,T,E L represents the total loss, α represents the first weight, β represents the second weight, y represents the true value, and p 学生,E’ p represents the fifth predicted probability value of the second labeled label belonging to the second label set E'. 学生,E p represents the third predicted probability value of the first labeled label belonging to the first label set E. 老师,T,E L1(y,p) represents the fourth predicted probability value of the first labeled label belonging to the first label set E. 学生,E’ L2(p) represents the first loss value. 学生,E ,p 老师,T,E The value represents the second loss value, and T represents the distillation temperature parameter that constrains the smoothness of the output of the fourth prediction probability value. For an introduction to the distillation temperature parameter T, please refer to the previous text; it will not be repeated here.

[0088] In one embodiment, L1() uses the cross-entropy loss function, and L2() uses the relative entropy loss function (KL divergence). In this case, the total loss function can be expressed as: αCE(y,p) 学生,E’ )+βKL(p 学生,E ,p 老师,T,E ), where CE is the cross-entropy loss function and KL is the KL divergence. Then, the student network model M is trained according to a certain training method (e.g., gradient descent training). 学生 The training is constrained by the total loss value until the student network model M is reached. 学生 If the preset conditions are met, such as the total loss value being less than a preset value, then the preset conditions are met; if it is greater than the preset value, then the preset conditions are not met. The calculation formulas for the cross-entropy loss function CE and KL divergence can be found in the corresponding formulas, and will not be explained in detail here.

[0089] By combining the above embodiments, the required student network model M can be trained. 学生 After obtaining the above student network model M 学生 Then, the student network model M can be used. 学生 The network identifies the sentences to be identified, thus obtaining the student network model M. 学生 The network outputs two values, and the final named entity recognition is performed based on these.

[0090] Obtain the student network model M 学生After being used as the target neural network model, the sentence to be identified can be input into the trained target neural network model to obtain the first predicted probability value of each first label in the first label set and the second predicted probability value of each second label in the second label set. Then, for each word, the label type of the first label with the highest first predicted probability value is taken as the first predicted label type of the word, and the label type of the second label with the highest second predicted probability value is taken as the second predicted label type of the word. That is, the predicted label is the label with the largest softmax value. In this way, two predicted label types can be obtained for each word. Finally, based on the first and second predicted label types of each word, the final predicted label type of the word is determined comprehensively, such as identifying whether it is label type O, label type B in the first label set, or label type B in the second label set, thereby realizing the named entity recognition task.

[0091] As can be seen, in this embodiment, the unique student network model M proposed in this application is used. 学生 Through this student network model M 学生 The two predicted label types are combined and output separately, making good use of the teacher's network model M. 老师 The predictive ability can be obtained without spending additional labeling time, effort, and training time to obtain the required student network model M. 学生 In entity label prediction, the lack of training on new datasets can lead to catastrophic forgetting issues. Therefore, the student network model M is used. 学生 Entities can also be used in the prediction output of new labels, which is highly practical and conducive to quickly establishing the required named entity recognition task.

[0092] In one embodiment, both the first tag set and the second tag set use BIO sequence labeling. In step S103, the final predicted tag type of the word is determined based on the first predicted tag type and the second predicted tag type, including:

[0093] S1030: When both the first predicted label type and the second predicted label type are label type O, the final predicted label type of the word is determined to be label type O.

[0094] It is understandable that if both the first and second predicted label types of a word are label type O, then the final predicted label type of the word is basically determined to be label type O, and there will be no other cases.

[0095] S1031: When one of the first predicted label type and the second predicted label type is label type B and the other is label type O, the final predicted label type of the word is determined to be label type B.

[0096] It is understandable that, since the labeled sequence of the predicted sentence is BII or similar, if the first linear layer predicts label type B, then there should be label type I later. It is impossible for the first label to be label type O and then be followed by label type I. Therefore, it can be determined that the final predicted label type of the word is label type B.

[0097] S1032: When both the first predicted label type and the second predicted label type are label type B, the label type with the highest predicted probability value is selected from the first predicted label type and the second predicted label type as the final predicted label type.

[0098] If both the first and second predicted label types are label type B, it means that both are identified as a certain named entity (such as a person's name and a place name). Then, the label type with the highest predicted probability value between the first and second predicted label types is taken as the final predicted label type. For example, the label type B with a higher predicted probability value for a person's name than for a place name is taken as the final label type.

[0099] S1033: If either the first predicted label type or the second predicted label type is label type I, then the final predicted label type is determined based on the entity label type and naming category of the preceding word.

[0100] If either the first predicted label type or the second predicted label type is label type I, then the final predicted label type needs to be determined based on the entity label type of the preceding word. That is, in one embodiment, the final predicted label type is determined based on the entity label type and entity naming category of the preceding word, including: if the label type of the preceding word is label type B or label type I, and the word naming category is the same as the entity naming category of the preceding word (e.g., both are personal names), then the final predicted label type is determined to be label type I; if the entity label type of the preceding word is label type O, or the entity naming category of the word is different from the entity naming category of the preceding word (e.g., one is a personal name, and the other is a place name), then one of the predicted label types previously determined to be label type I is changed to label type O (in S1033, it is first determined to be label type I and then changed to label type O), and the other predicted label type from the first and second predicted label types is combined to perform label prediction to determine the final predicted label type. The process of performing label prediction with the first predicted label type and the second predicted label type to determine the final predicted label type is the same as the process described in S1030-S1031 above, and will not be repeated here.

[0101] Understandably, if one of the predicted entity label types is label type I, then normally it should precede by B or I. Therefore, if the preceding word's label type is label type B or label type I, and the entity naming category of this word is the same as the preceding word's entity naming category (e.g., both are names), then the final predicted label type can be undoubtedly determined to be label type I. However, if the preceding word's entity label type is label type O, it indicates a prediction error, because such a label result is impossible in BIO labels. This means that the result of determining one of the predicted entity label types as label type I in the previous step is incorrect. In this embodiment, it is further changed to label type O, and then combined with the other predicted entity label type for comprehensive prediction, which effectively improves the prediction accuracy and effectiveness.

[0102] As can be seen, the above embodiments propose how to output the final predicted label type based on the two predicted label types output by the target neural network, thereby realizing the named entity recognition work.

[0103] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0104] In one embodiment, a named entity recognition device is provided, which corresponds one-to-one with the named entity recognition methods in the above embodiments. For example... Figure 4 As shown, the named entity recognition device 10 includes an acquisition module 101, an input module 102, a filtering module 103, and a determination module 104. Detailed descriptions of each functional module are as follows:

[0105] Module 101 retrieves the sentence to be recognized;

[0106] The input module 102 is used to input the sentence to be recognized into the trained target neural network model to obtain the first predicted probability value of each first label of the words in the sentence to be recognized belonging to the first label set, and the second predicted probability value of each second label of the words in the second label set.

[0107] The filtering module 103 is used to take the label type of the first label with the largest first prediction probability value as the first predicted label type of the word, and take the label type of the second label with the largest second prediction probability value as the second predicted label type of the word.

[0108] The determination module 104 is used to determine the final predicted label type of a word based on the first predicted label type and the second predicted label type of the word.

[0109] The target neural network model is obtained by distilling the student network model using a pre-trained named entity recognition model as the teacher network model.

[0110] In one embodiment, both the first tag set and the second tag set use BIO sequence labeling, and the determination module 104 is specifically used for:

[0111] If both the first and second predicted label types are label type O, then the final predicted label type of the word is determined to be label type O.

[0112] If one of the first predicted label type and the second predicted label type is label type B and the other is label type O, then the final predicted label type of the word is determined to be label type B.

[0113] When both the first predicted label type and the second predicted label type are label type B, the label type with the highest predicted probability value is selected as the final predicted label type from the first predicted label type and the second predicted label type.

[0114] If either the first predicted label type or the second predicted label type is label type I, then the final predicted label type is determined based on the label type of the preceding word and the entity naming category.

[0115] In one embodiment, the determining module 104 is further specifically used for:

[0116] If the label type of the preceding word is label type B or label type I, and the entity naming category of the word is the same as the entity naming category of the preceding word, then the final predicted label type is determined to be label type I.

[0117] If the label type of the preceding word is label type O, or if the naming category of the word is different from the entity naming category of the preceding word, then one of the predicted label types is changed to label type O, and a label prediction is performed with the other predicted label type among the first and second predicted label types to determine the final predicted label type.

[0118] In one embodiment, the target neural network model is trained as follows:

[0119] The training sentences are input into the pre-trained teacher network model to obtain the third predicted probability value of each word in the training sentences belonging to the first labeled label in the first label set;

[0120] The training sentences are input into the student network model to obtain the fourth prediction probability value of each word in the training sentences belonging to the first label and the fifth prediction probability value of each word belonging to the second label of the second label set.

[0121] The first loss value is calculated based on the fifth predicted probability value of each word and the true value of each word belonging to the second label, and the second loss value is calculated based on the third predicted probability value of each word belonging to the first label and the fourth predicted probability value of each word belonging to the first label.

[0122] The training of the student network model is constrained by the first loss value and the second loss value until the student network model meets the preset conditions. The student network model that meets the preset conditions is then used as the target neural network model.

[0123] In one embodiment, the named entity category corresponding to the entity annotation tag in the first tag set is different from the named entity category corresponding to the entity annotation tag in the second tag set.

[0124] In one embodiment, training the student network model is constrained according to a first loss value and a second loss value until the student network model meets preset conditions, including:

[0125] Obtain the first weight corresponding to the first loss value, and obtain the second weight corresponding to the second loss value;

[0126] Calculate the total loss value based on the first loss value, the second loss value, the first weight, and the second weight;

[0127] The training of the student network model is constrained by the total loss value until the student network model meets the preset conditions.

[0128] In one embodiment, based on a first loss value, a second loss value, a first weight, and a second weight,

[0129] Calculate the total loss value, including:

[0130] The total loss value is obtained using the following formula:

[0131] L=αL1(y,p 学生,E’ )+βL2(p 学生 ,E,p 老师,T,E );

[0132] L represents the total loss, α represents the first weight, β represents the second weight, y represents the true value, and p 学生,E’ p represents the fifth predicted probability value belonging to the second label. 学生,E p represents the third predicted probability value belonging to the first labeled item. 老师,T,E L1(y,p) represents the fourth predicted probability value belonging to the first labeled element. 学生,E’ L2(p) represents the first loss value. 学生,E ,p 老师,T,E ) represents the second loss value, and T represents the distillation temperature parameter that constrains the output smoothness of the fourth prediction probability value.

[0133] In one embodiment, the teacher network model includes a pre-trained first feature extraction network model, a first linear layer, and a first normalization layer. The output of the first feature extraction network model is connected to the input of the first linear layer, and the output of the first linear layer is connected to the input of the first normalization layer. The student network model includes a second feature extraction network model, a second linear layer, a second normalization layer, a third linear layer, and a third normalization layer. The output of the second feature extraction network model is connected to the inputs of the second linear layer and the third linear layer, respectively. The output of the second linear layer is connected to the input of the second normalization layer, and the output of the third linear layer is connected to the input of the third normalization layer.

[0134] In one embodiment, the initial training parameters of the second feature extraction network model are derived from the first feature extraction network model, and the initial training parameters of the second linear layer are derived from the first linear layer.

[0135] Specific limitations regarding the named entity recognition device can be found in the limitations of the named entity recognition method above, and will not be repeated here. Each module in the aforementioned named entity recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0136] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a readable storage medium and internal memory. The readable storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the readable storage medium. The database stores data related to the named entity recognition method. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a named entity recognition method. The readable storage medium provided in this embodiment includes both non-volatile and volatile readable storage media.

[0137] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0138] Obtain the recognized sentence;

[0139] The identified sentence is input into the trained target neural network model to obtain the first predicted probability value of each first label of the first label set and the second predicted probability value of each second label of the second label set.

[0140] The label type of the first label with the largest first predicted probability value is taken as the first predicted label type of the word, and the label type of the second label with the largest second predicted probability value is taken as the second predicted label type of the word;

[0141] Based on the first predicted label type and the second predicted label type of the word, determine the final predicted label type of the word;

[0142] The target neural network model is obtained by distilling the student network model using a pre-trained named entity recognition model as the teacher network model.

[0143] In one embodiment, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, performs the following steps:

[0144] Obtain the recognized sentence;

[0145] The identified sentence is input into the trained target neural network model to obtain the first predicted probability value of each first label of the first label set and the second predicted probability value of each second label of the second label set.

[0146] The label type of the first label with the largest first predicted probability value is taken as the first predicted label type of the word, and the label type of the second label with the largest second predicted probability value is taken as the second predicted label type of the word;

[0147] Based on the first predicted label type and the second predicted label type of the word, determine the final predicted label type of the word;

[0148] The target neural network model is obtained by distilling a pre-trained named entity recognition model as the teacher network model to train the student network model. In one embodiment, a computer program product is also provided, comprising a computer program that, when executed by a processor, performs the following steps:

[0149] Obtain the recognized sentence;

[0150] The identified sentence is input into the trained target neural network model to obtain the first predicted probability value of each first label of the first label set and the second predicted probability value of each second label of the second label set.

[0151] The label type of the first label with the largest first predicted probability value is taken as the first predicted label type of the word, and the label type of the second label with the largest second predicted probability value is taken as the second predicted label type of the word;

[0152] Based on the first predicted label type and the second predicted label type of the word, determine the final predicted label type of the word;

[0153] The target neural network model is obtained by distilling the student network model using a pre-trained named entity recognition model as the teacher network model.

[0154] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0155] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0156] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A named entity recognition method, characterized in that, The method includes: Obtain the recognized sentence; The identified sentence is input into the trained target neural network model to obtain the first predicted probability value of each first label of the first label set and the second predicted probability value of each second label of the second label set. The label type of the first label with the largest first predicted probability value is taken as the first predicted label type of the word, and the label type of the second label with the largest second predicted probability value is taken as the second predicted label type of the word; Based on the first predicted label type and the second predicted label type of the word, determine the final predicted label type of the word; The target neural network model is obtained by distilling the student network model using a pre-trained named entity recognition model as the teacher network model.

2. The named entity recognition method as described in claim 1, characterized in that, Each tag in the first and second tag sets is labeled using BIO sequence annotation. Determining the final predicted tag type of the word based on the first and second predicted tag types includes: When both the first predicted label type and the second predicted label type are label type O, the final predicted label type of the word is determined to be label type O. If one of the first predicted label type and the second predicted label type is label type B and the other is label type O, then the final predicted label type of the word is determined to be label type B. When both the first predicted label type and the second predicted label type are the label type B, the label type with the highest predicted probability value among the first predicted label type and the second predicted label type is taken as the final predicted label type. If either the first predicted label type or the second predicted label type is label type I, then the final predicted label type is determined based on the label type of the preceding word and the entity naming category.

3. The named entity recognition method as described in claim 2, characterized in that, The step of determining the final predicted label type based on the label type of the preceding word and the entity naming category includes: If the tag type of the preceding word is tag type B or tag type I, and the entity naming category of the word is the same as the entity naming category of the preceding word, then the final predicted tag type is determined to be tag type I. If the label type of the preceding word is label type O, or if the naming category of the word is different from the entity naming category of the preceding word, then one of the predicted label types is changed to label type O, and label prediction is performed by combining the first predicted label type and the other predicted label type from the second predicted label type to determine the final predicted label type.

4. The named entity recognition method as described in claim 1, characterized in that, The target neural network model is trained in the following manner: The training sentences are input into the pre-trained teacher network model to obtain the third predicted probability value of each word in the training sentences belonging to the first label in the first label set; The training sentence is input into the student network model to obtain the fourth prediction probability value of each word in the training sentence belonging to the first label and the fifth prediction probability value of each word belonging to the second label of the second label set. A first loss value is calculated based on the fifth predicted probability value of each word and the true value of each word belonging to the second label, and a second loss value is calculated based on the third predicted probability value of each word belonging to the first label and the fourth predicted probability value of each word belonging to the first label. The training of the student network model is constrained by the first loss value and the second loss value until the student network model meets the preset conditions. The student network model that meets the preset conditions is then used as the target neural network model.

5. The named entity recognition method as described in claim 4, characterized in that, The named entity categories corresponding to the entity annotation tags in the first tag set are different from the named entity categories corresponding to the entity annotation tags in the second tag set.

6. The named entity recognition method as described in claim 4, characterized in that, The step of constraining the training of the student network model based on the first loss value and the second loss value until the student network model meets preset conditions includes: Obtain the first weight corresponding to the first loss value, and obtain the second weight corresponding to the second loss value; Calculate the total loss value based on the first loss value, the second loss value, the first weight, and the second weight; The training of the student network model is constrained by the total loss value until the student network model meets the preset conditions.

7. The named entity recognition method as described in claim 6, characterized in that, The step of calculating the total loss value based on the first loss value, the second loss value, the first weight, and the second weight includes: The total loss value is obtained using the following formula: L=αL1(y, p 学生,E’ )+ βL2(p 学生,E , p 老师,T,E ); L represents the total loss value, α represents the first weight, β represents the second weight, y represents the true value, and p 学生,E’ p represents the fifth predicted probability value belonging to the second labeled item. 学生,E p represents the third predicted probability value belonging to the first labeled tag. 老师,T,E L1(y, p) represents the fourth predicted probability value belonging to the first labeled label. 学生,E’ L2(p) represents the first loss value. 学生,E , p 老师,T,E ) represents the second loss value, T represents the distillation temperature parameter that constrains the output smoothness of the fourth predicted probability value, E represents the first label set, and E' represents the second label set.

8. The named entity recognition method according to any one of claims 4-7, characterized in that, The teacher network model includes a pre-trained first feature extraction network model, a first linear layer, and a first normalization layer. The output of the first feature extraction network model is connected to the input of the first linear layer, and the output of the first linear layer is connected to the input of the first normalization layer. The student network model includes a second feature extraction network model, a second linear layer, a second normalization layer, a third linear layer, and a third normalization layer. The output of the second feature extraction network model is connected to the inputs of the second linear layer and the third linear layer, respectively. The output of the second linear layer is connected to the input of the second normalization layer, and the output of the third linear layer is connected to the input of the third normalization layer.

9. The named entity recognition method as described in claim 8, characterized in that, The initial training parameters of the second feature extraction network model are derived from the first feature extraction network model, and the initial training parameters of the second linear layer are derived from the first linear layer.

10. A named entity recognition device, characterized in that, The named entity recognition device includes: The acquisition module retrieves the sentence to be recognized. The input module is used to input the identified sentence into the trained target neural network model to obtain the first predicted probability value of each first label of the identified sentence belonging to the first label set and the second predicted probability value of each second label of the second label set. The filtering module is used to take the label type of the first label with the largest first predicted probability value as the first predicted label type of the word, and take the label type of the second label with the largest second predicted probability value as the second predicted label type of the word; The determination module is used to determine the final predicted label type of the word based on the first predicted label type and the second predicted label type of the word; The target neural network model is obtained by distilling the student network model using a pre-trained named entity recognition model as the teacher network model.

11. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 9.

12. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 9.

13. A computer program product, the computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 9.

Images