Chinese domain named entity recognition method fusing word vocabulary class representations

Abstract

The application relates to a Chinese field text named entity recognition method fusing word category representation, and belongs to the technical field of named entity recognition. The application sends character embedding, word embedding and category embedding into a Transformer encoder for coding and fusion, obtains category-enhanced character representation, combines the character representation and the category representation again through a gating mechanism at an output end, and sends the character representation and the category representation into CRF decoding to obtain entity labels of characters. Experiments are conducted on two data sets of a Chinese medical information processing challenge CBLUE, and results show that the performance of the application is obviously improved compared with other base models.

Description

TECHNICAL FIELD

[0001] The application relates to a Chinese field text named entity recognition method fusing word vocabulary category representation and belongs to the technical field of named entity recognition. BACKGROUND

[0002] Chinese named entity recognition mainly has a character-based or word-based sequence labeling method. The character-based method is to assign an entity label to each Chinese character, and the word-based method is to perform word segmentation on an input sentence first, and then assign an entity label to each word. In recent years, more Chinese named entity recognition often adopts the character-based method rather than the word-based method, because the word-based method will produce errors in the word segmentation process, so that the entity boundary cannot be correctly recognized. The character-based Chinese named entity recognition is affected by the lack of independent semantic expression ability of the Chinese character vector, which affects the model performance. Therefore, in recent years, researchers have introduced word information to improve the model performance. A typical one is the LatticeLSTM model, and the author constructs a Lattice structure to integrate word information into characters. However, LatticeLSTM can only interact the word with the last character of the word, and only relies on the information input of the previous step, but cannot use global information. The FLAT model later uses relative position coding to make characters and all matched words intersect, so as to better integrate word information into characters. On public data sets, these methods have achieved good results.

[0003] At present, the Chinese named entity recognition model mainly uses the character-based sequence labeling method, because the word-based method needs to perform word segmentation on the sentence. If errors occur in the word segmentation process, the model cannot correctly recognize the entity boundary. However, the character-based method only uses character information and cannot use global word information, so it is very important to integrate word information into the character-based Chinese named entity recognition method. In recent years, in order to solve the defects of the character-based method, the main research content of the general field Chinese named entity recognition is how to integrate word information into the character-based model. At present, the methods for improving the NER performance by fusing word information mainly have two routes, one is Dynamic Architecture, that is, a dynamic framework is designed to be compatible with word input. The other route is Adaptive Embedding, that is, based on word information, adaptive embedding is constructed. These two routes have their own advantages. Dynamic Architecture has better performance, but it is designed to integrate word information into each model, and cannot be conveniently transplanted. Adaptive Embedding has stronger portability, and the embedding mechanism is independent of the model and can be migrated to different models.

[0004] The model of Lattice LSTM proposed by Yang et al. first proposed the way of Adaptive Embedding. They fused the lexical information by constructing the Lattice format to improve the performance of NER. Huang et al. proposed Lexicon-Based CNNs and Refining Networks with Lexicon Rethinking, which solved the lexical conflict problem of Lattice LSTM. These two methods can only make the interaction between the word and the last character of the word, and only rely on the information input of the previous step, but cannot use global information, cannot capture long-distance dependencies, and have the problem of loss of lexical information. Liu et al. constructed a collaborative-based graph network, which consists of an encoding layer, a graph network layer, a fusion layer, and a decoding layer. In the graph network layer, the features of the first n character nodes in the graph network are extracted by Graph Attention Network (GAN). Huang et al. adopted lexicon-based graph neural network (LGN), which realizes the aggregation of local information through graph structure, takes each character as a node, forms edges by matching the vocabulary, and adds global nodes to integrate global information. In this way, the problems of Lattice LSTM are solved. These two methods need RNN as the underlying encoder to obtain global information, and the structure is relatively complex. Huang et al. flattened the Lattice structure, which was flattened from a directed acyclic graph to a flat Flat-Lattice Transformer structure. The interaction between the character and all the matching lexical information was modeled through relative position encoding, which could capture long-distance dependencies and improve the efficiency of the model compared with the previous method.

[0005] At present, the Chinese named entity recognition model based on the character-based method has made great progress in introducing lexical information, which can help the model to identify the boundary of the entity, but there is still a lack of judgment on the entity category. Therefore, the dictionary containing category information can make the model better identify the entity boundary and category.

[0006] In view of the above problems, the application provides a named entity recognition method fusing category information. On the disclosed CBLUE dataset, the application method is compared with a series of baseline methods, and the experimental results prove the effectiveness of the application method in the named entity recognition task. SUMMARY

[0007] In view of the above problems, the application provides a Chinese domain text named entity recognition method fusing word category representation, which explicitly embeds category information into word vectors, generates a fusion feature vector representation, and deepens the recognition ability of entity categories; the application uses category information to guide the extraction of entity categories, and to a certain extent, solves the problem that the model features in the previous related research work do not contain category information; and the performance of Chinese named entity recognition is improved.

[0008] The technical scheme of the application is a Chinese domain text named entity recognition method fusing word category representation, which comprises the following steps:

[0009] Step 1, obtaining a word vector representation based on text and its corresponding category vector representation;

[0010] Step 2, obtaining a fusion feature representation containing category information;

[0011] Step 3, inputting the fusion feature representation into a Transformer encoder;

[0012] Step 4, information filtering based on an interactive gating mechanism;

[0013] Step 5, CRF decoding to generate a predicted label.

[0014] As a further scheme of the application, in Step 1, a dictionary containing word category information is constructed, and the dictionary data is derived from the collected domain text and the training set information of the data set, and is stored in the format of "word, word category"; after obtaining the words contained in the text, the corresponding category is matched according to the dictionary, and the word vector and the category vector trained by Word2vec are obtained.

[0015] As a further scheme of the application, Step 1 specifically comprises the following:

[0016] For the input text sequence, the character vector The matched word vector And the category vector

[0017] For character embedding, the character vector trained on the corpus is used as the initial embedding of the character; the specific method is as follows: according to the trained character embedding lookup table, the embedding set trained by the Word2vec tool on the Chinese corpus Gigaword based on standard segmentation is used; thus, the character x i Is mapped to the character embedding representation And the bigram embedding representation

[0018]

[0019]

[0020] The field word embedding is trained using word2vec, with a dimension of 50, a window setting of 5, a minimum word frequency setting of 5, and a total of 5 training rounds; finally, the field word embedding index E is obtained w Each word y j is mapped to a word embedding representation

[0021]

[0022] For the category embedding of the word, a field dictionary L containing word categories is constructed t Stored in the format of "word, word category", constituting a field dictionary L containing word category information t ; L t The word y j is mapped to the category t j , and then mapped to an embedding representation through E w

[0023] t j = L t (y j ) (4)

[0024]

[0025] If the word does not find the corresponding category, it is mapped to a fixed randomly initialized embedding representation.

[0026] As a further scheme of the present application, in Step 2, after obtaining the category vector representation, it is spliced with the word vector to obtain a fusion feature representation containing category information.

[0027] As a further scheme of the present application, Step 2 specifically includes the following:

[0028] Step 2.1, for the input text sequence S = {x1, x2,...x m}, where x i represents the word in the sentence, the word and its category vector in the text are matched through different character spans, and the category representation is embedded into the word vector; the model matches the word y w in the dictionary E j ​​Put behind the sentence, at this time the text sequence becomes S1={x1,x2,...x n ,y1,y2,...y m};

[0029] Step2.2、Then, each character x i in the input sequence is mapped to a character embedding representation and a bigram feature embedding The and are spliced to obtain a character embedding fused with bigram features At this time, the input embedding sequence of the model is

[0030]

[0031] Wherein, denotes the concatenation operation of vectors;

[0032] Step2.3、For the words y j in the text sequence, each word in the input sequence is mapped to a word embedding and each word is mapped to a class embedding Then, the class vector of each word is concatenated with the word vector to obtain a word vector representation fused with class representation Let e denote the embedding, and the calculation process is as shown below.

[0033]

[0034] Step2.4、Finally, the vectors and are transformed into d model by linear transformation, where d represents the multi-head attention dimension, and the input vector sequence of the model at this time is

[0035] As a further scheme of the present application, in Step3, after obtaining the fused feature representation containing class information, the fused feature representation is input into the Transformer encoder to obtain character features fused with word and class information; the {B, M, E, O, S} marking scheme is adopted, wherein B represents the start position of the entity, M represents the middle position, E represents the end position, O represents the position without entity, and S represents an entity of a word.

[0036] As a further scheme of the present application, Step3 includes the following:

[0037] Two position encodings are added to each character and word, indicating the start and end positions of the word or character in the sentence, h i and t i represent the head and tail position coordinates of the character and word, respectively, and the distances of x i and y j are calculated from four different angles, thereby obtaining four relative distance matrices wherein represents the distance matrix between the head coordinates of x i and y j , represents the distance matrix between the tail coordinates of x i and the head coordinates of y j , represents the distance matrix between the head coordinates of x i and the tail coordinates of y j , represents the distance matrix between the tail coordinates of x i and y j ; after concatenating the four distances, a nonlinear transformation is performed to obtain the position encoding vector R i of x j and y ij , and the specific calculation process is shown in Equations 8-10.

[0038]

[0039]

[0040]

[0041] wherein d model represents the multi-head attention dimension, W r is a learnable parameter, represents the vector concatenation operation, d represents the four relative position matrices, and k represents the dimension index of the position encoding; the obtained input vector sequence E x is multiplied by the learnable parameter matrix W q , W k , and W v to obtain the attention weight matrix Q, K, and V, which are then subjected to calculation according to Equations 12 and 13.

[0042] [Q,K,V]=E x [W q ,W k ,W v ] (11)

[0043]

[0044] Att(Q,K,V)=softmax(A)V (13)

[0045] Among them W k,E W k,R u, v are learnable parameters, and A is the attention score. The next step is to calculate multi-head attention by mapping Q, K, and V through h different linear transformations, concatenating the different attention values ​​and performing another linear transformation. The calculation process is shown below:

[0046] Multi(Q,K,V)=Concat(head1,...,head h W m (14)

[0047]

[0048] Among them W m W i Q , It is a learnable parameter matrix, where h represents the number of attention heads. Finally, the Multi-head Attention result is input into the FFN layer to obtain the output character feature representation.

[0049]

[0050] Where z i It is the character vector output by the Multi-head Attention layer.

[0051] As a further embodiment of the present invention, in Step 4, after obtaining the character features of the fused words and category information, information enhancement is performed again and the contribution of character and category representations is balanced based on the interaction gating mechanism.

[0052] As a further aspect of the present invention, Step 4 includes the following:

[0053] Based on the vocabulary type corresponding to each character, the corresponding category representation and the character representation output by the encoder are concatenated; when a character corresponds to more than two types, the character x is concatenated. i The corresponding multiple words y j y j+1 ...the corresponding category representation is obtained through calculation. By concatenating them and performing a linear transformation, the corresponding category representation can be obtained. Then and character representation The splicing calculation formula is shown below;

[0054]

[0055] wherein represents the category representation of the character corresponding to the vocabulary, W t , b t is a learnable parameter, because the number of vocabularies corresponding to different characters is different, the number of vocabularies corresponding to characters in a sentence is counted, and the number of the most vocabularies corresponding to a character is l; for the characters whose vocabulary number is less than l, a fixed randomly initialized vector is supplemented to keep the dimension of all character vectors in the sentence consistent;

[0056] In order to dynamically balance the contribution of character representation and category representation, the proportion of category representation is controlled through an interactive gating mechanism, and the gating mechanism calculation formula is as follows.

[0057]

[0058]

[0059]

[0060] wherein σ represents a sigmod function, represents matrix element multiplication, G i is a gating output, W g , U g , b g is a learnable parameter.

[0061] As a further scheme of the present application, in Step 5, the character feature of the gating output is input into a CRF model to decode and obtain entity label information.

[0062] The beneficial effects of the present application are:

[0063] The present application first obtains a text-based word vector representation and its corresponding category vector representation; then, a fusion feature representation containing category information is obtained; secondly, the fusion feature representation is input into a Transformer encoder; information screening is performed based on an interactive gating mechanism; finally, a CRF decoding is performed to generate a predicted label. On the disclosed CBLUE dataset, the present application method is compared with a series of baseline methods, and the experimental results prove the effectiveness of the present application method in the named entity recognition task. BRIEF DESCRIPTION OF DRAWINGS

[0064] Fig. 1 The present application is a fusion category representation of the field text named entity recognition model framework;

[0065] Fig. 2 The present application is a category representation dimension ablation experiment result; DETAILED DESCRIPTION

[0066] So that the purposes, technical solutions and advantages of the embodiments of the present application are more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below with the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work belong to the protection scope of the present application. It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other at will.

[0067] Chinese named entity recognition is an important information extraction task. At present, the Chinese named entity recognition for domain text mainly adopts the sequence labeling method based on characters, and focuses on adding word information in the model to improve the performance of the model. Domain-specific vocabulary usually has corresponding entity categories, and the existing methods ignore the role of such category representation in entity boundary and category recognition.

[0068] The Chinese domain text named entity recognition method fusing vocabulary category representation provided by the present application will be described in detail below with reference to the drawings.

[0069] Embodiment 1: As shown in the figure, the Chinese domain text named entity recognition method fusing vocabulary category representation includes the following steps: Figs. 1-2

[0070] Step 1, obtaining the word vector representation based on the text and its corresponding category vector representation;

[0071] A dictionary containing vocabulary category information is constructed. The data used in the present application comes from the Chinese domain named entity recognition data sets CMeEE and CMeIE. The data set statistics are shown in Table 1.

[0072] Table 1 Data set statistics

[0073]

[0074] The dictionary data is stored in the format of "vocabulary, vocabulary category". After obtaining the vocabulary contained in the text, the corresponding category is matched according to the dictionary, and the word vector and the category vector trained by Word2vec are obtained. Specifically, the experimental environment of the present application is a deep learning framework Pytoch1.7.1 based on Python 3.7. The word vector and the category of the data set are initialized by using the Word2vec pre-training model, and the dimension of the word vector is set to d=50. The dimension of the hidden layer vector of the Transformer is d model ​Set to 160. During model training, set the learning rate for CMeIE to 1e-3, the learning rate for CMeEE to 2e-4, the batch size to 8, and use dropout to prevent overfitting.

[0075] Step 1 specifically includes the following:

[0076] For the input text sequence, obtain the character vectors respectively. Matched word vectors and its category vector

[0077] For character embedding, the character vectors trained on the corpus are used as the initial character embeddings. Specifically, based on the trained character embedding lookup table, the embedding set trained using the Word2vec tool is applied to the Gigaword Chinese corpus, which is based on standard word segmentation. Thus, the character x is embedded... i Mapping to character embedding representation and bigram embedding representation

[0078]

[0079]

[0080] Domain word embeddings were trained using word2vec with a dimension of 50, a window size of 5, and a minimum word frequency of 5, for a total of 5 training rounds; the final domain word embedding index E was obtained. w , for each word y j Mapping to a word embedding representation

[0081]

[0082] For the category embedding of words, a domain dictionary L containing word categories was constructed. t Stored in the format of "vocabulary, vocabulary category", forming a domain dictionary L containing vocabulary category information. t L t The word y j Mapping to category t j Then through E w Mapping to an embedded representation

[0083] t j =L t (y j (4)

[0084]

[0085] If no corresponding category is found for a word, it is mapped to a fixed, randomly initialized embedding representation.

[0086] Step 2: Obtain the fusion feature representation containing category information;

[0087] After obtaining the dictionary containing lexical category information from Step 1, in the input stage of the model, the category information is explicitly embedded into the word vectors by span matching the words in the text and their category feature vectors, thereby improving the named entity recognition task.

[0088] Step 2 specifically includes the following:

[0089] Step 2.1: For the input text sequence S = {x1, x2, ... x...} m}, where x i The model represents the characters in a sentence, matching words and their category vectors in the text with different character spans, and explicitly embedding the category representation into the word vectors; the model starts from the beginning of the sequence with different spans and a dictionary E. w The word matching process involves comparing words within the character set. If a word matches the character set, the matched word y is selected. j When placed after the sentence, the text sequence becomes S1 = {x1, x2, ... x}. n ,y1,y2,...y m};

[0090] Step 2.2: Then, use formulas (1) and (2) to process each character x in the input sequence. i Mapping to a character embedding representation and a bigram feature embedding Will and Concatenating the characters yields a character embedding that incorporates bigram features. At this point, the model's input embedding sequence is

[0091]

[0092] in, This indicates the operation of concatenating vectors.

[0093] Step 2.3: For the word y in the text sequence j Formula (3) maps each word in the input sequence to a lexical embedding. And each word is mapped to a category embedding using formulas (4) and (5). Then, the category vector of each word With vocabulary vectors concatenate the fusion class representation to get the word vector representation Let e denote the embedding, the calculation process is as follows:

[0094]

[0095] Step2.4、Finally, the vector and are changed to d model denotes the multi-head attention dimension, and the input vector sequence of the model at this time is

[0096] Step3, input the fusion feature representation into the Transformer encoder;

[0097] In Step3, after obtaining the fusion feature representation containing class information, the fusion feature representation is input into the Transformer encoder to obtain the character features of the fusion word and the class information; the {B, M, E, O, S} marking scheme is adopted, wherein B represents the starting position of the entity, M represents the middle position, E represents the end position, O represents the position without entity, and S represents an entity of a word.

[0098] As a further scheme of the application, Step3 includes the following:

[0099] Two position encodings are added to each character and word, respectively, to represent the starting and ending positions of the word or character in the sentence, as shown in the following formula: Fig. 1 h i and t i represent the head and tail position coordinates of the character and word, respectively, and the distances of x i and y j are calculated from four different angles, thereby obtaining four relative distance matrices wherein denotes the distance matrix between the head coordinates of x i and y j , denotes the distance matrix between the tail coordinates of x i and the head coordinates of y j , denotes the distance matrix between the head coordinates of x i and the tail coordinates of y j , denotes the distance matrix between the tail coordinates of x i and y j ; after concatenating the four distances, a nonlinear transformation is performed to obtain the position encoding vector R i of x j and y ij, the specific calculation process is shown in formulas 8-10;

[0100]

[0101]

[0102]

[0103] wherein d model represents the multi-head attention dimension, W r is a learnable parameter, represents the splicing operation of the vector, d represents four relative position matrices, and k represents the dimension index of the position encoding; the obtained input vector sequence E x is multiplied by the learnable parameter matrix W q , W k , W v to obtain the attention weight matrix Q, K, V, and then formula 12 and 13 are used for calculation;

[0104] [Q,K,V]=E x [W q ,W k ,W v ] (11)

[0105]

[0106] Att(Q,K,V)=softmax(A)V (13)

[0107] wherein W k,E , W k,R , u, and v are learnable parameters, and A is an attention score; the following calculation of multi-head attention is performed by mapping Q, K, and V through h different linear transformations, splicing different attentions, and performing a linear transformation again, and the calculation process is as follows:

[0108] Multi(Q,K,V)=Concat(head1,...,head h )W m (14)

[0109]

[0110] wherein W m , W i Q , is a learnable parameter matrix, and h represents the number of attention heads, and finally the multi-head attention result is input into the FFN layer to obtain the output character feature representation

[0111]

[0112] where z i is the character vector output by the Multi-head Attention layer.

[0113] Step4, information screening based on an interactive gating mechanism;

[0114] In Step4, after obtaining the character features fused with word and category information, information enhancement is performed again, and the contributions of character and category representations are balanced based on an interactive gating mechanism.

[0115] Step4 includes the following content:

[0116] According to the type of the word corresponding to each character, the corresponding category representation and the character representation output by the encoder are spliced; when there are more than two types corresponding to a character x i , the corresponding multiple words y j , y j+1 ... are calculated to obtain the corresponding category representation which is spliced and linearly transformed to obtain the corresponding category representation Then and the character representation are spliced, and the calculation formula is as follows:

[0117]

[0118] where represents the category representation of the word corresponding to the character, W t , b t are learnable parameters. Because the number of words corresponding to different characters is different, the number of words corresponding to characters in the sentence is counted, and the maximum number of words corresponding to a character is l; for characters with less than l words, a fixed randomly initialized vector is used to supplement, so that the dimensions of all character vectors in the sentence remain consistent.

[0119] In order to dynamically balance the contributions of character representation and category representation, the proportion of category representation is controlled through an interactive gating mechanism, and the gating mechanism calculation formula is as follows:

[0120]

[0121]

[0122]

[0123] where σ represents the sigmod function, ⊙ represents matrix multiplication, and Gi is the gated output, W g , U g , b g are learnable parameters.

[0124] Step 5, CRF decoding generates predicted labels. The character features of the gated output are input into the CRF model for decoding to obtain entity label information.

[0125] After obtaining the character features of the fusion class representation output in Step 4, the present application uses CRF as a decoder to obtain entity label information. Specifically, CRF is composed of an emission matrix and a transition matrix , where n is the number of characters in the sentence, and tags is the number of labels. The matrix E records the probability of each label, where E ij represents the probability that the i-th character is the j-th entity label; while the matrix T is a learnable parameter matrix, where T ij represents the probability of the j-th label transitioning to the i-th label, which is used to simulate the relationship between adjacent labels, and it can enable us to explicitly model the transition relationship between labels, thereby improving the accuracy of named entity recognition. The present application inputs the character features Z i of the fusion class representation into the CRF, and uses the negative log-likelihood function to calculate the label sequence with the maximum probability. The specific formula is as follows:

[0126] E = σ (W c h i + b c ) (21)

[0127]

[0128]

[0129] where W c , b c are learnable parameters, represents the sum of the emission probability between the observed sequence and the label sequence and the transition score of the label sequence, S represents the observed sequence, z represents the true label, and Z represents the set of true labels. Finally, we use the negative log-likelihood function to calculate the loss of the label:

[0130] Loss = -logp (z|S) (24)

[0131] In order to prove the effect of the present application, the fusion class information named entity recognition method proposed by the present application is compared with the baseline model, and the experimental results are shown in Table 2. The baseline model selected by the present application is mainly a Chinese named entity recognition model based on the word enhancement method, which is as follows:

[0132] Lattice LSTM: This model incorporates lexical information in a character-based sequence labeling framework, and constructs a Lattice format to represent the interaction between character information and lexical information.

[0133] LR-CNN: This model proposes Lexicon-Based CNNs and Refining Networks with Lexicon Rethinking to solve the lexical conflict problem of Lattice LSTM model.

[0134] LGN: This model adopts lexicon-based graph neural network (LGN), which aggregates local information through graph structure. Each character is taken as a node, and the matched vocabulary forms an edge. Global nodes are added to integrate global information. It solves the problem that Lattice LSTM only relies on the information input of the previous step and cannot utilize global information.

[0135] WC-LSTM: This model adopts Words Encoding Strategy to solve the problem that each character in Lattice LSTM can only obtain the number of words ending with it, which is dynamic and not fixed, resulting in the inability of Lattice LSTM to parallelize batch. Each character is represented by a fixed encoding to represent the information of the word ending with it. If there is no corresponding word, it is represented by <pad>denotes.

[0136] Simple-Lexicon: To avoid designing a complex model structure and facilitate migration to other sequence labeling frameworks, a method of simply using a lexicon in the embedding layer is proposed. For each character, all the lexicon sets corresponding to the four positions of BMES are obtained, and then encoded and represented.

[0137] FLAT: The model flattens the Lattice structure from a directed acyclic graph to a flat Flat-Lattice Transformer structure to directly model the interaction between characters and all matching lexical information, while capturing long-range dependencies.

[0138] MECT: The model proposes a Cross-Transformer to obtain the complementary semantic information of the structure of Chinese characters. Unlike the self-attention in the Transformer, the Cross-Transformer uses two Transformer encoders to cross the structure information of the Lattice and Chinese characters.

[0139] Table 2 compares the experimental results

[0140]

[0141]

[0142] Overall, as can be seen from Table 2, compared with all baseline models, the model TFLAT proposed in the present application achieves the highest F1 value on the CMeEE and CMeIE two datasets, verifying the effectiveness of the method of the present application. The model of the present application fuses word vectors and category vectors as feature representations, fully considers the category representation of the lexicon and the semantic information of the lexicon. Therefore, the model of the present application can obtain better performance. Compared with the LatticeLSTM and LR-CNN models which only rely on the information input of the previous step and cannot utilize global information, the model of the present application shows obvious advantages, and the performance is 3.0% and 3.5% higher than that of the LatticeLSTM on the two datasets. Compared with the LGN model and the WC-LSTM model, the model of the present application makes more sufficient use of lexical information and is more accurate for entity recognition, and the performance is 1.8% and 2.6% higher than that of the LGN model. Compared with the FLAT model, the model of the present application fuses the category representation, and the F1 value on the CMeEE and CMeIE two datasets is increased by 1.3% and 0.8%, respectively, verifying that the fusion of the category representation can help the model better identify entities.

[0143] In order to explore the importance of each part in the named entity recognition method of the model incorporating the category representation, the present application compares the use of CMeEE and CMeIE data sets in the following two experimental settings:

[0144] 1. w / o output-type: only incorporate vocabulary information and category representation at the input end to verify the improvement of model performance after the vocabulary information is fused with the category representation.

[0145] 2. w / o input-type: only fuse word features and category representations after the transformer encoder output features, without incorporating category representations at the input end, to verify the effect of fusing word features and category representations.

[0146] During the experiment, the experimental environment settings and steps 2 are consistent. The experimental results are shown in Table 3.

[0147] Table 3 Experimental results of different layers fusing category representations

[0148]

[0149] From the experimental results, it can be seen that when only incorporating category representations at the input end, the performance of the model on the CMeIE data set does not decrease significantly, and when only fusing word features and category representations after the transformer encoder output character features, the performance decreases significantly. On the CMeEE data set, it is found through experiments that the performance decreases significantly when only incorporating category representations at the input end and only fusing word information and category representations at the output end.

[0150] Through the above experiments and analysis, the present application considers that fusing word information and category representations at the input end and fusing word features and category representations at the output end both play an important role.

[0151] In order to further explore the influence of the dimension of the incorporated category representation on the performance of the model, the present application changes the dimension of the category vector through a linear transformation function on the CMeIE and CMeEE data sets, and compares different dimension vectors from 50 to 100, wherein 50 is the original dimension of the word vector and the word vector, and the remaining experimental settings remain the same. The experimental results are shown in Table 4. Fig. 2

[0152] From the experimental results, it can be seen that the category vector with a dimension of 50 has the best effect, and the performance decreases when the vector dimension is expanded. This may be due to the fact that the category vector dimension is too large, reducing the information ratio of the word vector and the word vector, resulting in a decrease in the information of the word and the word obtained by the model, thereby affecting the performance of the model.

[0153] ​Through the above experiments and analysis, the application considers that using a category vector with a dimension of 50, that is, keeping the category vector dimension consistent with the word vector and the word vector dimension, is the most appropriate.

[0154] The application proposes a method for Chinese domain text named entity recognition fusing category information. The application uses category information to guide the extraction of entity categories, and to a certain extent, solves the problem that the model features in the previous related research work do not contain category information. Firstly, the vocabulary and category information are fused in the model input stage to deepen the model's understanding of data semantics; then, after the transformer encoder outputs the features, the corresponding category vectors are spliced according to the category of the word corresponding to each word to further strengthen the category information. Experiments are conducted on two data sets of CALUA, and the experimental results show that the performance of the text proposed method model fusing category information in the Chinese domain text named entity recognition task is better than that of the remaining baseline models, thereby verifying that fusing category information can improve the performance of the Chinese named entity recognition model.

[0155] The specific embodiments of the application are described in detail above with reference to the accompanying drawings, but the application is not limited to the above-described embodiments, and various changes can be made within the knowledge of those skilled in the art without departing from the purpose of the application.< / pad>

Claims

1. A Chinese domain text named entity recognition method based on fusion of vocabulary category representation, characterized in that: The Chinese domain text named entity recognition method represented by the fusion word category includes the following steps: Step 1, obtaining a word vector representation based on text and its corresponding category vector representation; Step 2, obtaining a fusion feature representation containing category information; Step 3, inputting the fusion feature representation into a Transformer encoder; Step 4, information screening based on an interactive gating mechanism; Step 5, CRF decoding to generate a predicted label; The Step 2 specifically includes the following: Step 2.1, for the input text sequence wherein represents a word in a sentence, by matching the words in the text and their category vectors through different character spans, the category representation is explicitly embedded into the word vector; The model starts from the beginning of the sequence with different spans and dictionaries. The word matching process involves comparing words within the character set. If a word matches the character set, the matched word is... When placed after a sentence, the text sequence becomes... ; Step 2.

2. Then, each character in the input sequence is mapped to a character embedding representation and a bigram feature embedding , and the two are concatenated to obtain a character embedding that incorporates bigram features , i.e. and , to obtain a character embedding that incorporates bigram features , and the input embedding sequence of the model at this time is ; ； wherein denotes a concatenation operation of vectors; Step 2.3, for each word in the text sequence mapping each word in the input sequence to a word embedding and mapping each word to a class embedding ; Subsequently, the category vector of each vocabulary is obtained and the vocabulary vector is spliced to obtain a vocabulary vector representation of the fusion category representation , where e represents embedding, and the calculation process is shown as follows; ； Step 2.4, finally, the vector and is transformed into , , denotes the multi-head attention dimension, where the input vector sequence of the model is .

2. The Chinese domain text named entity recognition method of claim 1, wherein: In the Step 1, a dictionary containing word category information is constructed, and the dictionary data is derived from the collected domain text and the training set information of the data set, and is stored in the format of "word, word category"; after obtaining the words contained in the text, the corresponding category is matched according to the dictionary, and the word vector and the category vector trained by Word2vec are obtained.

3. The Chinese domain text named entity recognition method of claim 1, wherein: The Step 1 specifically includes the following: For the input text sequence, respectively obtain a character vector , a matching vocabulary vector , and a category vector ; For character embeddings, use the character vectors trained on the corpus as the initial embedding for the character; specifically, map characters to character embedding representations and bigram embedding representations using the embedding set trained on the Gigaword corpus using the Word2vec tool based on the standard segmented Chinese corpus. ； ； The domain word embedding is trained using word2vec with a dimension of 50, a window setting of 5, a minimum word frequency setting of 5, and a total of 5 training rounds; finally, a domain word embedding index is obtained Each vocabulary word is mapped to a word embedding representation :​ ； For lexical class embedding, a domain lexicon containing lexical classes is constructed , stored in the format "lexical, lexical class", and forms a domain lexicon containing lexical class information ; Lexical is mapped to class , and then mapped to an embedding representation :​ ； ； If the word cannot find the corresponding category, it will be mapped to a fixed random initialized embedding representation.

4. The Chinese domain text named entity recognition method of claim 1, wherein: In the Step 2, after obtaining the category vector representation, it is spliced with the word vector to obtain a fusion feature representation containing category information.

5. The Chinese domain text named entity recognition method of claim 1, wherein: In Step 3, after obtaining the fusion feature representation containing category information, the fusion feature representation is input into a Transformer encoder to obtain character features of the fusion word and the category information; and a labeling scheme, wherein a start position of an entity is represented by an intermediate position is represented by an end position is represented by a position without an entity is represented by an entity of one word.

6. The Chinese domain text named entity recognition method of claim 1, wherein: The Step 3 includes the following: Two position encodings are added to each character and word, indicating the start and end positions of the word or character in the sentence, and respectively, and the distances between the head and tail coordinates of the characters and words are calculated from four different angles and , resulting in four relative distance matrices , , , wherein denotes the distance matrix between the head coordinates of and , denotes the distance matrix between the tail coordinates of and the head coordinates of , denotes the distance matrix between the head coordinates of and the tail coordinates of , denotes the distance matrix between the tail coordinates of and ; the four distances are concatenated and subjected to a non-linear transformation, resulting in the position encoding vectors and , the specific calculation process is shown in equations 8-10;​ ； ； ； wherein, represents the multi-head attention dimension, is a learnable parameter, represents the concatenation operation of vectors, d represents four relative position matrices, and k represents the dimension index of the position encoding; the obtained input vector sequence is multiplied by the learnable parameter matrix to obtain the attention weight matrix , and then formula 12 and 13 are used for calculation. ； ； ； in , , , These are learnable parameters. This is the attention score; the next step is to calculate multi-head attention, through... Different linear transformations The mapping process involves concatenating the different Attention values ​​and then performing a linear transformation. The calculation process is shown below: ； ； wherein , , , is a learnable parameter matrix, denotes the number of attention heads, and the Multi-headAttention result is finally input into the FFN layer to obtain the output character feature representation ; ； wherein is the character vector output of the Multi-head Attention layer.

7. The Chinese domain text named entity recognition method of fusing vocabulary category representation according to claim 1, characterized in that: In the Step 4, after obtaining the character features of the fusion word and category information, information enhancement is performed again, and the contributions of character and category representation are balanced based on the interactive gating mechanism.

8. The Chinese domain text named entity recognition method of claim 1, wherein: The Step 4 includes the following content: According to the type of the vocabulary corresponding to each character, the character representation output by the corresponding category representation and the encoder is spliced; when the type corresponding to a character is more than two, the character corresponding to multiple vocabularies , ... the corresponding category representation is obtained by calculation , ... which is spliced and linearly transformed to obtain the corresponding category representation , and and the character representation are spliced, and the calculation formula is as follows; ； wherein represents a category representation of the vocabulary corresponding to the character, , is a learnable parameter, because the number of vocabularies corresponding to different characters is different, the number of vocabularies corresponding to the character in the sentence is counted, and the number of the most vocabularies corresponding to one character is l; for the character whose vocabulary number is less than l, a fixed randomly initialized vector is supplemented to keep the dimension of all character vectors in the sentence consistent; In order to dynamically balance the contributions of character representation and category representation, the interactive gating mechanism is used to control the proportion of category representation, and the gating mechanism calculation formula is as follows: ； ； ； wherein represents a sigmod function, represents an element-wise multiplication, is a gated output, , , are learnable parameters.

9. The Chinese domain text named entity recognition method of claim 1, wherein: In the Step 5, the character features output by the gating are input into the CRF model for decoding to obtain entity label information.

Images