A method and device for named entity recognition

Abstract

The present application provides a method and device for named entity recognition, which adds prior knowledge of at least one field to sample data; based on the prior knowledge of at least one field, the pre-training model is trained, and the number of multi-head attention mechanisms in the pre-training model is reduced, and the number of multi-head attention mechanisms is consistent with the field of named entity recognition. For example, in the named entity recognition in the film and television field, the distinction in terms of part of speech mainly focuses on the recognition fields such as nouns, verbs, adjectives, and numerals. Let different head attention mechanisms focus on different recognition fields, and at the same time, widen the distance between different head attention mechanisms, so that the semantic expressions learned by the multi-head attention mechanism do not produce redundancy.

Description

Technical Field

[0001] The present application belongs to the field of natural language processing technology, and in particular, relates to a method and device for named entity recognition. Background Art

[0002] Named Entity refers to entity words with specific meaning or strong representativeness in a certain field, such as event names, place names, and person names in the news field. Named Entity Recognition (NER) refers to identifying entities with specific meanings in text, such as identifying names of people, places, institutions, proper nouns, etc.

[0003] As an important task in the field of Natural Language Processing (NLP), named entity recognition has gradually abandoned the use of recurrent neural networks (RNNs) for named entity recognition with the development of pre-trained models. Instead, named entity recognition has begun to be based on pre-trained models. Pre-trained models can include transform, bert, electra and other models. Compared with RNN recognition, pre-trained models can improve model effects (such as recognition accuracy). However, due to the large number of parameters in pre-trained models, model performance (such as reasoning speed) is reduced, which makes it difficult to implement pre-trained models. Therefore, the performance of the model is improved by losing the model effect during the training process of the pre-trained model.

[0004] For example, one way is to perform named entity recognition based on the BERT model. The BERT model fine-tunes the sample training set to reduce the number of sentences in the sample training set and annotate the sentences to obtain the semantic expression features of the characters (tokens) in the sentences, and perform named entity recognition based on the semantic expression features. The BERT model uses the attention mechanism to obtain the semantic expression features of the token. For example, the BERT model reduces the number of heads of the multi-head attention mechanism and uses a 12-head attention mechanism to obtain the semantic expression features of sentences in different semantic fields. For example, semantics includes parts of speech, morphology, syntax, word meaning and other fields. The 12-head attention mechanism can be used to obtain the semantic expression features in these fields.

[0005] After the number of heads of the Multi-head attention mechanism in the BERT model is reduced, the number of parameters of the BERT model is also reduced, which can improve the model performance. However, reducing the number of heads of the Multi-head attention mechanism may cause redundancy in the acquired semantic expression features, that is, the semantic expressions learned by the attention mechanisms of different heads are similar or overlapped, which reduces the model effect. Summary of the invention

[0006] In view of this, the purpose of the present application is to provide a method and device for named entity recognition.

[0007] In a first aspect, the present application provides a method for named entity recognition, the method comprising:

[0008] Add prior knowledge of at least one field to the sample data;

[0009] Based on the prior knowledge of at least one field, the pre-trained model is trained, and during the training process of the pre-trained model, the distance between different probability distributions obtained by the pre-trained model is increased and the number of parameters of the pre-trained model is reduced. The pre-trained model is used for named entity recognition.

[0010] Optionally, reducing the number of parameters of the pre-trained model includes: reducing the number of heads of a multi-head attention mechanism in the pre-trained model.

[0011] Optionally, reducing the number of heads of the multi-head attention mechanism in the pre-trained model includes: setting the number of heads of the multi-head attention mechanism based on the number of recognition fields targeted by the pre-trained model, the number of heads of the multi-head attention mechanism being the same as the number of recognition fields.

[0012] Optionally, the training of the pre-trained model based on the prior knowledge of the at least one field includes:

[0013] Perform word mapping on each sentence in the input text to obtain a semantic expression matrix;

[0014] Performing position mapping on each sentence in the input text to obtain a position expression matrix;

[0015] Based on the prior knowledge of the at least one field, annotating each sentence in the input text to obtain an annotation matrix;

[0016] The pre-training model is trained based on the semantic expression matrix, the position expression matrix and the annotation matrix.

[0017] Optionally, the training of the pre-trained model based on the semantic expression matrix, the position expression matrix and the annotation matrix includes:

[0018] Based on the semantic expression matrix and the position expression matrix, obtaining first input data;

[0019] Processing the first input data based on the first network of the pre-trained model to obtain a loss value of the first network, wherein the first network can increase the distance between different probability distributions obtained by the first network;

[0020] Based at least on the labeling matrix, obtain second input data and third input data;

[0021] Processing the second input data based on the second network of the pre-trained model to obtain a loss value of the second network;

[0022] Processing the third input data based on the third network of the pre-trained model to obtain a loss value of the third network, wherein the second input data and the third input data are different, and / or the second network and the third network are different;

[0023] Based on the loss value of the first network, the loss value of the second network, and the loss value of the third network, the model parameters of the pre-trained model are adjusted until the pre-trained model converges.

[0024] Optionally, the first network includes a multi-head attention mechanism, and the first network based on the pre-trained model processes the first input data to obtain a loss value of the first network, including:

[0025] Processing the first input data based on the first network to obtain an output of each attention mechanism;

[0026] Get the distance between the outputs of any two attention mechanisms;

[0027] Based on the distance between the outputs of any two attention mechanisms, a loss value of the first network is obtained, and the loss value of the first network has the characteristic of increasing the distance between the different probability distributions obtained by the any two attention mechanisms, and the different probability distributions obtained by the any two attention mechanisms are the outputs of the any two attention mechanisms.

[0028] Optionally, adding prior knowledge of at least one field to the sample data includes: adding prior knowledge of part-of-speech information to the sample data.

[0029] Optionally, the method further includes:

[0030] Based on the preset dictionary, the predicted sentence is mapped to the domain to obtain the first domain feature matrix;

[0031] The first domain feature matrix and the sentence to be predicted are input into the pre-trained model obtained by training to obtain an entity recognition result of the sentence to be predicted output by the pre-trained model.

[0032] In a second aspect, the present application provides a named entity recognition device, the device comprising:

[0033] A preprocessing unit, used for adding prior knowledge of at least one field to the sample data;

[0034] A training unit is used to train the pre-trained model based on prior knowledge of at least one field, increase the distance between different probability distributions obtained by the pre-trained model and reduce the number of parameters of the pre-trained model during the training process of the pre-trained model, and the pre-trained model is used for named entity recognition.

[0035] In a third aspect, the present application provides an electronic device, the electronic device comprising:

[0036] One or more processors, memory;

[0037] The memory is used to store computer program codes, and the computer program codes include computer instructions. When the one or more processors execute the computer instructions, the electronic device executes the above-mentioned named entity recognition method.

[0038] In a fourth aspect, the present application provides a computer storage medium for storing a computer program, which, when executed, is specifically used to implement the above-mentioned named entity recognition method.

[0039] During the training process of the above-mentioned pre-trained model, prior knowledge of at least one field is added to increase the model's learning ability in this field, assist the pre-trained model in predicting in this field, and improve the accuracy. For example, prior knowledge of part-of-speech information can be added so that the pre-trained model can be more focused on predicting the part-of-speech field, thereby improving the accuracy of part-of-speech recognition. In addition, the distance between the probability distributions obtained by the model is increased, and the similarity or overlap between the semantic expression features learned by different processing layers in the pre-trained model is reduced, so that there is no redundancy or reduced redundancy between the semantic expression features learned by the pre-trained model. In addition, the number of parameters of the pre-trained model is reduced, and the model performance is improved by reducing the number of parameters. However, because there is no redundancy or reduced redundancy between the semantic expression features learned by the pre-trained model, the model effect can be improved while the model performance is improved.

[0040] For example, the pre-trained model can be a BERT model. During the training of the BERT model, in order to reduce the redundancy or lack of redundancy between the semantic expression features learned by the multi-head attention mechanism, distance calculation is used to increase the distance between the probability distributions output by the multi-head attention mechanism. When the multi-head attention mechanism can fully learn the semantic expression features in different sub-fields, the redundancy or lack of redundancy between the semantic expression features learned by the multi-head attention mechanism can also be reduced. In addition, reducing the number of heads of the multi-head attention mechanism can improve the accuracy while speeding up the recognition rate of the BERT model. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 It is a training flow chart of a pre-training model provided in an embodiment of the present application;

[0043] Figure 2 This is a flowchart of a named entity recognition based on a pre-trained model provided in an embodiment of the present application;

[0044] Figure 3 It is a structural diagram of a named entity recognition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0046] In this application, the pre-trained model can be used for named entity recognition, for example, named entity recognition in the film and television field based on the pre-trained model. During the training process of the pre-trained model, the pre-trained model is optimized as follows:

[0047] 1) Add prior knowledge of at least one field to the sample data to increase the model's learning ability in that field;

[0048] 2) Increasing the distance between the probability distributions obtained by the model, reducing the similarity or overlap between the semantic expression features learned by different processing layers in the pre-training model, so that there is no redundancy or the redundancy is reduced between the semantic expression features learned by the pre-training model;

[0049] 3) Reduce the number of parameters of the pre-trained model. Improve the model performance by reducing the number of parameters. However, because there is no redundancy or reduced redundancy between the semantic expression features learned by the pre-trained model, the model effect can be improved while improving the model performance.

[0050] For example, the pre-trained model can be a BERT model. In the process of training the BERT model, the prior knowledge of part-of-speech information is added to increase the learning ability of the BERT model in terms of part-of-speech. Of course, other types of prior knowledge can also be added in the process of training the BERT model. For example, if the task of named entity recognition is related to syntax, the prior knowledge of syntactic information can be added.

[0051] For the BERT model, reduce the number of heads of the multi-head attention mechanism, such as reducing the number of heads of the multi-head attention mechanism to 4-6. Generally, the number of heads of the multi-head attention mechanism in pre-trained models such as BERT is related to the field of named entity recognition. For example, in the NER task in the film and television field, the four sub-fields of nouns, numerals, adjectives, and verbs in the part-of-speech field are concerned. The number of heads of the multi-head attention mechanism in the BERT model is 4, and different attention mechanisms target different sub-fields. In order to reduce the redundancy or lack of redundancy between the semantic expression features learned by the multi-head attention mechanism, distance calculation is used to increase the distance between the probability distributions output by the multi-head attention mechanism. When the multi-head attention mechanism can fully learn the semantic expression features in different sub-fields, the redundancy or lack of redundancy between the semantic expression features learned by the multi-head attention mechanism can also be reduced.

[0052] The following is an explanation of the training process of the pre-trained model provided in this application with reference to the accompanying drawings. Figure 1 , which shows the training process of the pre-trained model provided by the present application, which may include the following steps:

[0053] S101. Perform word embedding on each sentence in the input text to obtain a semantic expression matrix. The semantic expression matrix is ​​used to indicate the semantic expression features of each word in each sentence. The semantic expression features of each sentence can be a vector, and the semantic expression matrix is ​​composed of the vectors of each sentence. The input text can be a preset number of sentences selected from a sample data set. The preset number is the number of sentences processed by the pre-trained model at one time, which can be set according to demand, for example, 10 sentences are processed at one time.

[0054] For example, a sentence Q = {w 1 ,w 2 ,...,w i ,...,w L},w iis the i-th word in sentence Q, and L is the length of the sentence. In this embodiment, sentence Q is mapped using a randomly initialized word embedding method, and a semantic expression matrix is ​​obtained after word embedding is completed for a preset number of sentences. The semantic expression feature can be recorded as M, and the dimension size of M is {batch_size, seq_len, dim}, where batch_size is a preset number, seq_len is the length of the sentence, and dim is the dimension of word embedding, and the dimension of word embedding can be preset. The length of all sentences in the input text can be the same, for example, if the length of all sentences is L, then seq_len = L; if the lengths of all sentences are different, seq_len is the length of the longest sentence.

[0055] S102, position mapping (position embedding) is performed on each sentence in the input text to obtain a position expression matrix. The position expression matrix is ​​used to indicate the position features of each word in each sentence. The position expression feature of each sentence can be a vector, and the position expression matrix is ​​composed of the vectors of each sentence. For example, a randomly initialized position embedding method is used to map sentence Q. After completing position embedding for a preset number of sentences, a position expression matrix is ​​obtained.

[0056] The position expression matrix can be denoted as P. The dimension of the position expression matrix P can be the same as the dimension of the semantic expression matrix M. Of course, the dimension of the position expression matrix P can be different from the dimension of the semantic expression matrix M. For example, the dimension of position embedding is different from the dimension of word embedding, which makes the dimension of the position expression matrix P different from the dimension of the semantic expression matrix M.

[0057] S103, perform part-of-speech tagging on each sentence in the input text to obtain a part-of-speech expression matrix. The part-of-speech expression matrix is ​​used to indicate the part-of-speech feature of each word in each sentence. The part-of-speech expression feature of each sentence can be a vector, and the part-of-speech expression matrix is ​​formed by the vectors of each sentence. The part-of-speech expression matrix can be denoted as F, and the dimension of the part-of-speech expression matrix F can be different from the dimension of the semantic expression matrix M, depending on the part-of-speech tagging.

[0058] Because the part-of-speech expression matrix can record the part-of-speech features of each word in each sentence, it is equivalent to adding prior knowledge of part-of-speech information. Of course, this embodiment can also add prior knowledge of other types of information, and the type of prior knowledge added is related to the recognition field of the pre-trained model. If the recognition field is part-of-speech, the prior knowledge of part-of-speech information is added; if the recognition field is syntax, the prior knowledge of syntactic information is added.

[0059] For example, the BIEO (Begin-Intermediate-End-Other) method is used to annotate the characters in a sentence. When using the BIEO method to annotate, the preset dictionary can be used for annotation, and the preset dictionary records the part-of-speech category of each character. For example, the part-of-speech category in the preset dictionary can be related to the part-of-speech field that the pre-trained model focuses on. For example, the part-of-speech field that the pre-trained model focuses on includes: nouns, numerals, adjectives, and verbs. The part-of-speech categories in the preset dictionary include: nouns, numerals, adjectives, and verbs. The annotation forms of the characters under these four part-of-speech categories are recorded in the preset dictionary.

[0060] One way is to use different characters to identify the part-of-speech category, for example, the part-of-speech category of nouns is 1, the part-of-speech category of numerals is 2, the part-of-speech category of adjectives is 3, and the part-of-speech category of verbs is 4, thereby distinguishing the parts of speech from 1 to 4. When using the BIEO method to mark, the beginning mark is B-TYPE, the end mark is E-TYPE, and the mark without relevant part-of-speech is O. For example, the verb "play" has its part-of-speech mark B-4 and E-4; if the part of speech of a word is not a noun, numeral, adjective, or verb, it is marked as O.

[0061] In this embodiment, subcategories can also be added to a part-of-speech category. For example, subcategories such as artist, film name, and theater can be added to the noun category, and the subcategories can be marked in the dictionary. For example, "Liu XY's WuXX", Liu XY is artist, and the three words Liu XY are marked as B-artist, I-artist, and E-artist; WuXX is name, and the three words WuXX are marked as B-name, I-name, and E-name. The corresponding total number of part-of-speech categories is the total number of all subcategories.

[0062] S104: Obtain first input data of the pre-trained model, and process the first input data based on a first network of the pre-trained model to obtain an output of the first network.

[0063] The first input data is the input of the first network. The first network is trained for semantic expression and position expression. The first input data can be obtained based on the semantic expression matrix M and the position expression matrix P.

[0064] One example is that the first input data H is obtained by fusion of the semantic expression matrix M and the position expression matrix P. One way is to concatenate the semantic expression matrix M and the position expression matrix P to obtain the first input data H; another way is to add the semantic expression matrix M and the position expression matrix P element by element to obtain the first input data H.

[0065] The network architecture of the first network may be: the first network includes N layers of transformers, the number of transformer layers depends on the situation, for example, N may be 6 or 12. The input of the first layer of transformers in the first network is the first input data, the input of the remaining transformers is the output of the previous layer of transformers, and the output of the last layer of transformers is taken as the output of the first network.

[0066] The structure of each transformer layer can be the same. One transformer structure is that the transformer includes a multi-head attention mechanism, two layers of residual networks and a fully connected feedforward network. The fully connected feedforward network can use a feedforward neural network (FFN). The relationship is that the multi-head attention mechanism is connected to a residual network, and the fully connected feedforward network is connected between the two residual networks. The input H of the i-th transformer layer i After the multi-head attention mechanism, we get Input H i And the multi-head attention mechanism Input together into the residual network connected with the multi-head attention mechanism, we get The calculation formula can be:

[0067] As the input of the fully connected feedforward network, we get The calculation formula can be W and b are the model parameters of FFN.

[0068] and The final output H is obtained by inputting them together into the residual network connected to the fully connected feedforward network. i+1 .H i+1 The calculation formula is Get the input H of the i+1th layer transformer i+1 By analogy, after N layers of transformers, the output of the first network is H n .

[0069] The processing of the multi-head attention mechanism can be Q h , K h and V h is the element in the matrix, d k is the dimension size of the matrix. MultiH(X)=[H1 , …, H |h| ] is the output of the multi-head attention mechanism, and h is the number of individual heads.

[0070] S105. After obtaining the output of each transformer, calculate the distance between any two outputs. Based on the distance between any two outputs, obtain the loss value of the first network.

[0071] In order to allow different attention mechanisms to learn semantic expressions in different recognition fields, the distance between the outputs of the transformers corresponding to different attention mechanisms is as far as possible to avoid more redundancy in learning. Based on this, this embodiment can use KL-divergence to calculate the distance between any two outputs.

[0072] Assume that the outputs of any two attention mechanisms in the last layer of transformer are H ni and H nj , respectively represent the probability distribution of the output of the two attention mechanisms, and H is calculated based on the following formula ni and H nj Distance between:

[0073]

[0074] D KL (H ni ||H nj ) represents the distance between two probability distributions, and n is the total number of elements in the output of the attention mechanism, such as H ni The total number of elements in p(x i ) is H ni The i-th element in q(x i ) is H nj The i-th element in .

[0075] After calculating the distance between any two outputs, all distances are weighted averaged to obtain the distance of the first network, such as Y is the number of heads of the multi-head attention mechanism, which is related to the recognition field. For example, Y = 4, and entity recognition is performed for 4 recognition fields.

[0076] Where γ is the maximum distance of KL divergence in the multi-head attention mechanism, which is a hyperparameter and is set to 1 in this embodiment. Loss1 is to find the maximum value of each sentence in the input text and then add it up. If a sentence If the value is greater than γ, the loss of the sentence is 0, otherwise the loss of the sentence is

[0077] S106: Obtain second input data of the pre-trained model, process the second input data based on a second network of the pre-trained model, and obtain a loss value of the second network.

[0078] The second input data is the input of the second network. The second network is trained at least for part-of-speech expression, so the second input data can be obtained based on at least the part-of-speech expression matrix F. The second input data E can be the part-of-speech expression matrix F, or it can be a fusion of the semantic expression matrix M, the position expression matrix P and the part-of-speech expression matrix F. For the fusion method, please refer to the description of the first input data H.

[0079] The second network may have a different network structure from the first network. In one example, part of the network structure of the second network is the same as the network structure of the first network. For example, the second network also includes multiple N-layer transformers, and the second input data is output after passing through the N-layer transformer. n , for H n Process again to obtain the part-of-speech tagging matrix E.

[0080] For example, for H n The entity annotation matrix E is obtained by the fully connected feedforward neural network transformation. One form is that the entity annotation matrix E is obtained by softmax processing after the linear transformation of the fully connected layer, such as based on the formula E = softmax (H n W E ), W E It is a parameter used in the training process of the pre-trained model, and softmax is a normalization function.

[0081] Based on the part-of-speech tagging results in the part-of-speech tagging matrix E and the part-of-speech expression matrix F, the loss value of the second network is obtained using the cross entropy method. The calculation formula is as follows:

[0082] Where n is the total number of parts of speech, p is i is the part-of-speech tagging result in the part-of-speech expression matrix F, q i is the part-of-speech tagging result in the part-of-speech tagging matrix E.

[0083] S107: Obtain third input data of the pre-trained model, process the third input data based on the third network of the pre-trained model, and obtain a loss value of the third network.

[0084] The third network may be trained at least for part-of-speech expressions, and the third input data may be obtained based on the part-of-speech expression matrix F. Because the second network and the third network are both trained for part-of-speech expressions, the network structures of the second network and the third network may be different, or the second input data and the third input data may be different.

[0085] In this embodiment, the third input data may be obtained by merging the semantic expression matrix M and the part-of-speech expression matrix F. For the fusion method, please refer to the description of the first input data H. The third network may be a bidirectional RNN. The output H of the bidirectional RNN network s First, it undergoes a linear transformation in the fully connected layer, and then undergoes softmax processing to obtain the part-of-speech matrix P. The corresponding calculation formula is P = softmax (H s W p ), W p are the parameters used during the training of the pre-trained model.

[0086] Based on the part-of-speech tagging results in the part-of-speech matrix P and the part-of-speech expression matrix F, the loss value of the third network is obtained using the cross entropy method. The calculation formula is as follows:

[0087] Where n is the total number of parts of speech, p is i is the part-of-speech tagging result in the part-of-speech expression matrix F, q i is the part-of-speech tagging result in the part-of-speech matrix P.

[0088] S108. Adjust the model parameters of the pre-trained model based on the loss value of the first network, the loss value of the second network, and the loss value of the third network.

[0089] Based on the loss value of the first network, the loss value of the second network and the loss value of the third network, the loss value of the pre-trained model is obtained; based on the loss value of the pre-trained model, the model parameters of the pre-trained model are adjusted. The loss value of the pre-trained model Loss = αLoss1 + βLoss2 + γLoss3, where α, β, γ are hyperparameters with a value range of (0-1), and their values ​​can be determined according to actual conditions. When adjusting the model parameters based on the loss value of the pre-trained model, the gradient descent algorithm can be used to adjust the model parameters. After the model parameters are adjusted, repeat the above steps S101 to S108 until the loss value of the model converges and the model training ends.

[0090] During the training process of the above-mentioned pre-trained model, prior knowledge of at least one field is added to increase the model's learning ability in this field, assist the pre-trained model in predicting in this field, and improve the accuracy. For example, prior knowledge of part-of-speech information can be added so that the pre-trained model can be more focused on predicting the part-of-speech field, thereby improving the accuracy of part-of-speech recognition. In addition, the distance between the probability distributions obtained by the model is increased, and the similarity or overlap between the semantic expression features learned by different processing layers in the pre-trained model is reduced, so that there is no redundancy or reduced redundancy between the semantic expression features learned by the pre-trained model. In addition, the number of parameters of the pre-trained model is reduced, and the model performance is improved by reducing the number of parameters. However, because there is no redundancy or reduced redundancy between the semantic expression features learned by the pre-trained model, the model effect can be improved while the model performance is improved.

[0091] For example, the pre-trained model can be a BERT model. During the training of the BERT model, in order to reduce the redundancy or lack of redundancy between the semantic expression features learned by the multi-head attention mechanism, distance calculation is used to increase the distance between the probability distributions output by the multi-head attention mechanism. When the multi-head attention mechanism can fully learn the semantic expression features in different sub-fields, the redundancy or lack of redundancy between the semantic expression features learned by the multi-head attention mechanism can also be reduced. In addition, reducing the number of heads of the multi-head attention mechanism can improve the accuracy while speeding up the recognition rate of the BERT model.

[0092] After completing the training of the pre-trained model, you can use the pre-trained model to perform named entity recognition. The process is as follows: Figure 2 As shown, the following steps may be included:

[0093] S201: Based on a preset dictionary, perform part-of-speech mapping on the sentence to be predicted to obtain a first part-of-speech feature matrix. For example, the dictionary in step S103 may be used to perform part-of-speech tagging to obtain the first part-of-speech feature matrix.

[0094] S202: Input the first part-of-speech feature matrix and the sentence to be predicted into the pre-trained model, obtain the entity recognition result of the sentence to be predicted output by the pre-trained model, and complete the part-of-speech recognition of the sentence to be predicted, for example, by using the second network in the pre-trained model.

[0095] Refer to the above Figure 1 and Figure 2The method for named entity recognition provided in this embodiment may include: adding prior knowledge of at least one field to sample data; training the pre-training model based on the prior knowledge of at least one field, increasing the distance between different probability distributions obtained by the pre-training model and reducing the number of parameters of the pre-training model during the training of the pre-training model, and the pre-training model is used for named entity recognition. For example, prior knowledge of part-of-speech information is added.

[0096] Optionally, reducing the number of parameters of the pre-trained model includes: reducing the number of heads of the multi-head attention mechanism in the pre-trained model. For example, reducing the number of heads of the multi-head attention mechanism in the pre-trained model includes: setting the number of heads of the multi-head attention mechanism based on the number of recognition fields targeted by the pre-trained model, and the number of heads of the multi-head attention mechanism is the same as the number of recognition fields.

[0097] Optionally, the training of the pre-trained model based on the prior knowledge of at least one field includes: performing character mapping on each sentence in the input text to obtain a semantic expression matrix; performing position mapping on each sentence in the input text to obtain a position expression matrix; annotating each sentence in the input text based on the prior knowledge of at least one field to obtain an annotation matrix; and training the pre-trained model based on the semantic expression matrix, the position expression matrix, and the annotation matrix. The annotation matrix can be the above-mentioned part-of-speech expression matrix. For the methods of obtaining different matrices and the model training process, please refer to Figure 1 shown.

[0098] Optionally, the training of the pre-trained model based on the semantic expression matrix, the position expression matrix and the annotation matrix includes: obtaining first input data based on the semantic expression matrix and the position expression matrix; processing the first input data based on the first network of the pre-trained model to obtain a loss value of the first network, wherein the first network can increase the distance between different probability distributions obtained by the first network; obtaining second input data and third input data based at least on the annotation matrix; processing the second input data based on the second network of the pre-trained model to obtain a loss value of the second network; processing the third input data based on the third network of the pre-trained model to obtain a loss value of the third network, wherein the second input data and the third input data are different, and / or the second network and the third network are different; adjusting the model parameters of the pre-trained model based on the loss value of the first network, the loss value of the second network and the loss value of the third network until the pre-trained model converges.

[0099] Optionally, the first network includes a multi-head attention mechanism, and the first network based on the pre-trained model processes the first input data to obtain the loss value of the first network, including: processing the first input data based on the first network to obtain the output of each attention mechanism; obtaining the distance between the outputs of any two attention mechanisms; and obtaining the loss value of the first network based on the distance between the outputs of any two attention mechanisms, and the loss value of the first network has the characteristic of increasing the distance between different probability distributions obtained by the any two attention mechanisms, and the different probability distributions obtained by the any two attention mechanisms are the outputs of the any two attention mechanisms.

[0100] In this embodiment, the process from the first network to the third network, from the first input data to the third input data, and the loss value calculation is shown in the above Figure 1 Description of the illustrated embodiment.

[0101] Optionally, adding prior knowledge of at least one field to the sample data includes: adding prior knowledge of part-of-speech information to the sample data.

[0102] Optionally, the method also includes: based on a preset dictionary, performing domain mapping on the sentence to be predicted to obtain a first domain feature matrix; inputting the first domain feature matrix and the sentence to be predicted into a trained pre-trained model to obtain an entity recognition result of the sentence to be predicted output by the pre-trained model.

[0103] For example, to identify the part of speech of the sentence to be predicted, based on the preset dictionary, the sentence to be predicted is mapped to the part of speech to obtain a first part of speech feature matrix, such as using the dictionary in step S103 to perform part of speech tagging to obtain the first part of speech feature matrix. The first part of speech feature matrix and the sentence to be predicted are input into the pre-trained model to obtain the part of speech recognition result of the sentence to be predicted output by the pre-trained model.

[0104] In addition, the present application provides a named entity recognition device, whose structure is as follows: Figure 3 As shown, it may include: a preprocessing unit 10 and a training unit 20.

[0105] The preprocessing unit 10 is used to add prior knowledge of at least one field to the sample data; for example, adding prior knowledge of at least one field to the sample data includes: adding prior knowledge of part-of-speech information to the sample data.

[0106] The training unit 20 is used to train the pre-trained model based on the prior knowledge of at least one field, increase the distance between different probability distributions obtained by the pre-trained model and reduce the number of parameters of the pre-trained model during the training process of the pre-trained model, and the pre-trained model is used for named entity recognition.

[0107] Optionally, the training unit 20 may reduce the number of heads of the multi-head attention mechanism in the pre-trained model. For example, the training unit 20 sets the number of heads of the multi-head attention mechanism based on the number of recognition fields targeted by the pre-trained model, and the number of heads of the multi-head attention mechanism is the same as the number of recognition fields.

[0108] Optionally, the training unit 20 trains the pre-trained model based on the prior knowledge of at least one field, including: performing character mapping on each sentence in the input text to obtain a semantic expression matrix; performing position mapping on each sentence in the input text to obtain a position expression matrix; based on the prior knowledge of at least one field, annotating each sentence in the input text to obtain an annotation matrix; and training the pre-trained model based on the semantic expression matrix, the position expression matrix and the annotation matrix.

[0109] Optionally, the training unit 20 trains the pre-trained model based on the semantic expression matrix, the position expression matrix and the annotation matrix, including: obtaining first input data based on the semantic expression matrix and the position expression matrix; processing the first input data based on the first network of the pre-trained model to obtain a loss value of the first network, wherein the first network can increase the distance between different probability distributions obtained by the first network; obtaining second input data and third input data based at least on the annotation matrix; processing the second input data based on the second network of the pre-trained model to obtain a loss value of the second network; processing the third input data based on the third network of the pre-trained model to obtain a loss value of the third network, wherein the second input data and the third input data are different, and / or the second network and the third network are different; adjusting the model parameters of the pre-trained model based on the loss value of the first network, the loss value of the second network and the loss value of the third network until the pre-trained model converges.

[0110] Optionally, the first network includes a multi-head attention mechanism, and the training unit 20 processes the first input data based on the first network of the pre-trained model to obtain the loss value of the first network, including: processing the first input data based on the first network to obtain the output of each attention mechanism; obtaining the distance between the outputs of any two attention mechanisms; and obtaining the loss value of the first network based on the distance between the outputs of any two attention mechanisms, wherein the loss value of the first network has the characteristic of increasing the distance between different probability distributions obtained by the any two attention mechanisms, and the different probability distributions obtained by the any two attention mechanisms are the outputs of the any two attention mechanisms.

[0111] Optionally, the named entity recognition device also includes: a prediction unit, which is used to perform domain mapping on the sentence to be predicted based on a preset dictionary to obtain a first domain feature matrix; input the first domain feature matrix and the sentence to be predicted into a pre-trained model obtained by training to obtain an entity recognition result of the sentence to be predicted output by the pre-trained model.

[0112] For example, to identify the part of speech of the sentence to be predicted, based on the preset dictionary, the sentence to be predicted is mapped to the part of speech to obtain a first part of speech feature matrix, such as using the dictionary in step S103 to perform part of speech tagging to obtain the first part of speech feature matrix. The first part of speech feature matrix and the sentence to be predicted are input into the pre-trained model to obtain the part of speech recognition result of the sentence to be predicted output by the pre-trained model.

[0113] In addition, the present application also provides an electronic device, which includes: one or more processors and a memory; the memory is used to store computer program code, and the computer program code includes computer instructions. When the one or more processors execute the computer instructions, the electronic device executes the above-mentioned named entity recognition method.

[0114] The present application also provides a computer storage medium for storing a computer program, which, when executed, is specifically used to implement the above-mentioned named entity recognition method.

[0115] It should be noted that the various embodiments in this specification can be described in a progressive manner, and the features recorded in the various embodiments in this specification can be replaced or combined with each other. Each embodiment focuses on the differences from other embodiments, and the same and similar parts between the various embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0116] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0117] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0118] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for named entity recognition, characterized in that: The method comprises: Add prior knowledge of at least one field to the sample data; Based on sample data to which prior knowledge of at least one field is added, a pre-trained model is trained, and during the training process of the pre-trained model, the distance between different probability distributions obtained by the pre-trained model is increased and the number of parameters of the pre-trained model is reduced, wherein the pre-trained model is used for named entity recognition; The training of the pre-trained model based on the sample data to which the prior knowledge of at least one field is added comprises: Perform word mapping on each sentence in the input text to obtain a semantic expression matrix; Performing position mapping on each sentence in the input text to obtain a position expression matrix; Based on the prior knowledge of the at least one field, annotating each sentence in the input text to obtain an annotation matrix; The pre-training model is trained based on the semantic expression matrix, the position expression matrix and the annotation matrix.

2. The method according to claim 1, characterized in that The reducing the number of parameters of the pre-trained model includes: reducing the number of heads of the multi-head attention mechanism in the pre-trained model; Among them, reducing the number of heads of the multi-head attention mechanism in the pre-trained model includes: setting the number of heads of the multi-head attention mechanism based on the number of recognition fields targeted by the pre-trained model, and the number of heads of the multi-head attention mechanism is the same as the number of recognition fields.

3. The method according to claim 1, characterized in that The training of the pre-trained model based on the semantic expression matrix, the position expression matrix and the annotation matrix includes: Based on the semantic expression matrix and the position expression matrix, obtaining first input data; Processing the first input data based on the first network of the pre-trained model to obtain a loss value of the first network, wherein the first network can increase the distance between different probability distributions obtained by the first network; Based at least on the labeling matrix, obtain second input data and third input data; Processing the second input data based on the second network of the pre-trained model to obtain a loss value of the second network; Processing the third input data based on the third network of the pre-trained model to obtain a loss value of the third network, wherein the second input data and the third input data are different, and / or the second network and the third network are different; Based on the loss value of the first network, the loss value of the second network, and the loss value of the third network, the model parameters of the pre-trained model are adjusted until the pre-trained model converges.

4. The method according to claim 3, characterized in that The first network includes a multi-head attention mechanism, and the first network based on the pre-trained model processes the first input data to obtain a loss value of the first network including: Processing the first input data based on the first network to obtain an output of each attention mechanism; Get the distance between the outputs of any two attention mechanisms; Based on the distance between the outputs of any two attention mechanisms, a loss value of the first network is obtained, and the loss value of the first network has the characteristic of increasing the distance between the different probability distributions obtained by the any two attention mechanisms, and the different probability distributions obtained by the any two attention mechanisms are the outputs of the any two attention mechanisms.

5. The method according to any one of claims 1 to 4, characterized in that: The adding of prior knowledge of at least one field into the sample data includes: adding prior knowledge of part-of-speech information into the sample data.

6. The method according to claim 5, characterized in that The method further comprises: Based on the preset dictionary, the predicted sentence is mapped to the domain to obtain the first domain feature matrix; The first domain feature matrix and the sentence to be predicted are input into the pre-trained model obtained by training to obtain an entity recognition result of the sentence to be predicted output by the pre-trained model.

7. A named entity recognition device, characterized in that: The device comprises: A preprocessing unit, used for adding prior knowledge of at least one field to the sample data; A training unit, configured to train a pre-trained model based on sample data to which prior knowledge of at least one field is added, and to increase the distance between different probability distributions obtained by the pre-trained model and reduce the number of parameters of the pre-trained model during the training of the pre-trained model, wherein the pre-trained model is used for named entity recognition; The training unit is specifically used for: Perform word mapping on each sentence in the input text to obtain a semantic expression matrix; Performing position mapping on each sentence in the input text to obtain a position expression matrix; Based on the prior knowledge of the at least one field, annotating each sentence in the input text to obtain an annotation matrix; The pre-training model is trained based on the semantic expression matrix, the position expression matrix and the annotation matrix.

8. An electronic device, characterized in that: The electronic device comprises: One or more processors, memory; The memory is used to store computer program code, which includes computer instructions. When the one or more processors execute the computer instructions, the electronic device executes the named entity recognition method as described in any one of claims 1 to 6.

9. A computer storage medium, characterized in that: Used to store a computer program, which, when executed, is specifically used to implement the named entity recognition method as described in any one of claims 1 to 6.

Images