Chinese named entity recognition method based on stacked grid structure information enhancement

Abstract

The invention discloses a Chinese named entity recognition method based on stacked grid structure information enhancement, and belongs to the technical field of natural language processing and named entity recognition. A stacked grid structure and a hierarchical compressed sensing module are used for introduction and lossless compression of word vectors, a long input sequence and information loss are avoided, and the recognition efficiency is improved. Meanwhile, the vocabulary fusion process and the context feature coding process are decoupled to reduce unnecessary attention calculation, and the calculation efficiency is improved; font structure information of Chinese characters is introduced as a supplement, the relevance of entities on multiple granularities of fonts, characters and words is mined through multiple cross attention, and the context representation ability of a text is enhanced; an entity recognition method based on duplex collaboration is used, collaborative learning between span extraction and label classification is achieved through soft label constraint and a multi-task joint scoring function, label classification training is completed through full samples, error propagation caused by span extraction errors is reduced, and the recognition accuracy is improved.

Description

Technical Field

[0001] The invention belongs to the technical field of natural language processing and named entity recognition, and in particular to a Chinese named entity recognition method based on stacked grid structure information enhancement. Background Art

[0002] In recent years, deep learning technology has been widely used in the field of named entity recognition. Huang et al. used a bidirectional long short-term memory network Bi-LSTM as an encoder to learn the contextual representation of characters and used a conditional random field (CRF) as a decoder to extract entities. Cao et al. proposed a model based on a convolutional neural network (CNN) and used iterative dilated convolution as an encoder to extract features, effectively avoiding the overfitting problem caused by too many convolution layers. These methods all focus on improving the feature extraction capability at the character level. When processing text, they often only focus on single characters or character sequences and ignore the information of vocabulary or vocabulary sequences. However, in the task of named entity recognition, this vocabulary information is usually very important. For example, a complete entity name may consist of multiple words, and character-based methods may not be able to accurately identify such entities. The word-based named entity recognition method generally performs word segmentation first and then performs entity tagging on the obtained word sequence. This method will affect the subsequent entity tagging process due to word segmentation errors. In this regard, many methods have shown that character-based methods are superior to word-based methods.

[0003] In Chinese named entity recognition, Zhang and Yang first proposed a Chinese named entity recognition model based on vocabulary information enhancement—Lattice LSTM. The character-word lattice structure used in this model encodes and combines character sequences with potential words matched by the dictionary, avoiding error propagation caused by word segmentation errors and improving the performance of Chinese named entity recognition. However, due to the inconsistent number of words fused to each character, it cannot be parallelized, resulting in poor computing performance. In addition, due to the structure of LSTM itself, the model can only rely on the information input of the previous step and cannot use global information. Gui et al. proposed the LR-CNN model, which uses CNN to integrate vocabulary information to achieve parallel processing, and the introduced rethinking mechanism can consider global vocabulary information. Sui and Gui et al. proposed a vocabulary enhancement method based on a graph network model, using graph neural networks to achieve character-word information aggregation. These methods are faster than Lattice-LSTM, but are still computationally complex and not transferable. Qiu et al. proposed the FLAT model based on the Transformer structure, which captures long-distance dependencies through the self-attention mechanism and solves the problems of parallel computing and global dependencies. The PLTE model proposed by Mengge et al. also uses Transformer as an encoder and enhances local modeling capabilities through a porous mechanism. Although these subsequent vocabulary enhancement methods solve the problem of parallel computing in the process of dictionary introduction and model global vocabulary information with the attention mechanism, they will generate long input sequences when processing long texts due to matching more words, resulting in the need for sequence truncation or heuristic selection of some words in practical applications. Both methods will cause information loss. Therefore, it is necessary to design a new vocabulary enhancement method to overcome the limitations of existing methods.

[0004] In addition, some researchers have also tried to introduce information such as glyphs and pronunciations to improve the text representation capabilities of Chinese named entity recognition models. Dong et al. first introduced glyph information into Chinese named entity recognition, extracted glyph embeddings through Bi-LSTM and combined them with character embeddings. Cui Shaoguo et al. fused Chinese character image features and Wubi font encoding for named entity recognition of Chinese electronic medical record data. Mai et al. combined Chinese character phonetic features with dictionary information to help eliminate entity boundary ambiguity. These methods only use information such as glyphs and pronunciations as supplementary information for character features, without considering the relevance of entities at these different granularities. Summary of the invention

[0005] The present invention solves the problem of information loss caused by the existing word information enhancement method based on character-word lattice structure when processing long texts, in which the input sequence grows due to the excessive number of words, forcing the model to trim the input sequence or filter some words. The present invention also enhances the text representation capability and improves the entity recognition accuracy by fusing the semantic information of entities at multiple granularities (grapheme, character and vocabulary).

[0006] The present invention is a Chinese named entity recognition method based on stacked grid structure information enhancement. First, in a word fusion module based on stacked grids, through stacked grid structures and hierarchical compressed sensing, the compression and fusion of word vectors are realized while accommodating more vocabulary vectors. This process not only retains all vocabulary information, but also avoids information loss. And by decoupling vocabulary fusion and context feature encoding processes, unnecessary character-character and word-word attention calculations in the fusion process are reduced. Then, considering the relevance of entities at three different granularities of glyphs, characters and words, a multi-granularity fusion module is proposed, which captures the relevance features of entities at multiple granularities through multiple cross-attention mechanisms to enhance the context representation ability of entities. Finally, an entity recognition module based on dual-link collaboration is proposed, which realizes multi-task collaborative optimization by constructing soft label constraints between candidate entities and the final entity prediction labels, and relies on the application of full samples in label classification tasks to avoid error propagation caused by entity discrimination errors and improve the accuracy of entity recognition. Compared with the above-mentioned other inventions, the present invention can better enhance entity boundaries through word information and make full use of multi-granularity information (glyphs, characters and words) to improve the vector representation ability of text without information loss, and further improve the entity recognition accuracy through the proposed named entity recognition module. Thus, it shows better performance in evaluation indicators such as accuracy and recall.

[0007] To achieve the above purpose, the technical solution adopted by the present invention is:

[0008] A Chinese named entity recognition method based on stacked grid structure information enhancement includes a word fusion module based on stacked grids, a multi-granularity fusion module based on cross-attention, and an entity recognition module based on double-link collaboration. The specific method includes the following steps:

[0009] Step S1: In a word fusion module based on a stacked grid, a stacked grid structure is constructed to store all word vectors that match the characters in the sentence, and the word vector information is first compressed using hierarchical compressed sensing and then the attention mechanism is used to realize the vector fusion of characters and words;

[0010] Step S2: In a multi-granularity fusion module based on cross-attention, multiple cross-attentions are used to jointly learn the fused word vector sequence, the glyph sequence vector obtained by the glyph embedding method, and the character vector sequence to capture the correlation characteristics of the entity at multiple granularities;

[0011] Step S3: Identify the entity to be extracted and determine the category to which it belongs through the entity recognition module based on dual collaboration.

[0012] The further improvement of the technical solution of the present invention is that the specific steps of step S1 are:

[0013] Step S1.1 uses character embedding and word embedding on the sentence to obtain a character vector sequence of the sentence and a word vector sequence to be matched;

[0014] Step S1.2 builds a multi-layer stacked grid structure to accommodate the word vector sequence;

[0015] Step S1.3 applies hierarchical compressed sensing to compress the word vector information layer by layer, and uses multi-head attention using rotational position encoding to achieve vector fusion between characters and words.

[0016] The further improvement of the technical solution of the present invention is that the specific method of step S1.1 is:

[0017] Given a sentence X = (x1, x2, ..., x n ), where n is the length of the sentence, and the character vector sequence representation H of sentence X is obtained by embedding the context words:

[0018] H=(h1,...,h n )=Embedding(x1,...,x n ) (1)

[0019] Where Embedding() means using the contextual word embedding of the pre-trained model BERT, h i is the vector representation of a single character;

[0020] For sentence X, use the dictionary tree w_T to match the characters in the sentence one by one, obtain the word sequence, and then use the word vector to look up the table R w Get the corresponding word vector sequence in i s and i e Respectively represent the starting and ending positions of words in a sentence. is the vector representation of the word, and m is the number of words;

[0021] The further improvement of the technical solution of the present invention is that the specific method of step S1.2 is:

[0022] After obtaining the word vector sequence W of the sentence o After that, a stacked grid structure is constructed, whose dimension size is R L*L*L , where L is the maximum length of sentences in the current batch of samples; the maximum number of words that this structure can accommodate is The multi-layer stacking mode adopted by this structure enables the word vectors to be aligned one by one with the character vector subsequences in the sequence, which facilitates the subsequent word vector compression process.

[0023] A further improvement of the technical solution of the present invention is that the specific method of step S1.3 is:

[0024] The word vector is compressed. The compression process includes internal compression and boundary compression. Internal compression is to compress the word vector layer by layer and distribute the compressed word vector to all the constituent characters of the word. Boundary compression compresses the word vector information to the left and right boundaries of the word. The compressed word vector is used to enhance the first and last boundaries of the word. The vectors obtained by internal compression and boundary compression are merged and fused with the character vector under the guidance of the multi-head attention mechanism. The fused word vector contains the pairwise matching information between characters and words and the global feature information between character sequences and word sequences.

[0025] A further improvement of the technical solution of the present invention is that: the specific method of internal compression is: first initialize a unit vector sequence U=(u1,...,u n ), and then U is combined with the word vector sequence in the stacked grid structure Perform one-by-one matching calculations:

[0026] W i F =u i W i G (2)

[0027] Get a character-by-character compressed vector

[0028] The specific method of boundary compression is: perform head and tail boundary matching calculation on the unit vector sequence and the word vector sequence after superimposing the multi-layer grid structure:

[0029]

[0030] Get the compressed head and tail boundary word vector W S and W E ;

[0031] W F , W S and W EThe word sequence H obtained by concatenation W The query matrix Q, key matrix K, and value matrix V for attention calculation are obtained by linear transformation with the sentence character vector sequence H:

[0032] [Q,K,V]=[Hω q ,H w ω k ,H w ω v ] (5)

[0033] where ω q ,ω k and ω v are all learnable parameter matrices.

[0034] A further improvement of the technical solution of the present invention is that position coding is introduced to retain the position information of the input sequence during attention calculation, and rotational position coding RoPE is used as the position coding. Specifically, the method of adding position coding to q∈Q and k∈K is as follows:

[0035] q m =f(q,m)=R f (q,m)e iΘf(q,m) =||q||e i(Θ(q)+mθ) =qe imθ (6)

[0036] k n =f(k,n)=R f (k,n)e iΘf(k,n) =||k||e i(Θ(k)+nθ) =ke inθ (7)

[0037] in The transformed vector is and Then use attention calculation to get the fused word vector W :

[0038]

[0039] where u is a learnable parameter of the attention bias.

[0040] The further improvement of the technical solution of the present invention is that the specific steps of step S2 are:

[0041] For sentence X, firstly, the glyph sequence of each word in the sentence is obtained by the word splitting dictionary P, and then the glyph sequence is converted into a glyph vector sequence by applying the glyph vector lookup table R. Finally, the glyph vector sequence is feature extracted using the dilated convolution DConv to obtain the glyph embedding representation G:

[0042] G=(g1,g2,...,g n )=DConv(R(P(X))) (9)

[0043] For sentence X, its corresponding character sequence vector is H, the fused word sequence vector is W and the glyph vector sequence is G, where the cross attention between the character vector and the word vector is calculated as follows:

[0044] Head (i) =CrossAtt(((Hω q ) (i) +u) Τ (Wω k ) (i) ,(Wω v ) (i) ) (10)

[0045] MultiHead(H,W)=[Head (1) ,Head (2) ,...,Head (l) ] (11)

[0046] where ω q ,ω k and ω v They are all learnable parameter matrices, u is the attention bias, and l is the number of heads in the multi-head cross attention;

[0047] The calculation of the cross attention between the character vector and the glyph vector is the same as the above calculation. The character-word fusion feature vector H1 and the character-glyph fusion feature vector H2 obtained by this module are input into a fully connected layer for information fusion to obtain the fusion vector C:

[0048]

[0049] in represents the connection operation, W o and b are learnable parameters.

[0050] A further improvement of the technical solution of the present invention is that: step S3 is based on a dual-cooperative entity recognition method, and an entity scoring function and a label scoring function are introduced in the span extraction and label classification tasks respectively, which work together on the final entity recognition. The joint loss function of the soft label constraint is combined to realize the collaborative processing between span extraction and label classification, and the full amount of samples are used in the label classification task. In order to improve the label classification effect in the multi-classification task, the same classification strategy as that in the forward unlabeled learning model is adopted in the label classification task. By constructing a multi-head label classifier, the recognition accuracy of the method in each category is improved.

[0051] A further improvement of the technical solution of the present invention is that for sentence X, it is necessary to enumerate all possible seed spans S = (s0, s1, ..., s m ), m is the number of seed spans of the sentence, and each span is represented by s = (x i ,x j ), 1≤i≤j≤n, where i and j represent the positions of the left and right boundaries of the span in the sentence, respectively, and n is the length of the sentence;

[0052] The fusion vector C obtained by the multi-granularity fusion module is sent to this module to extract candidate entities. At this time, the span vector is represented by s = (c i ,c j ), the span score is calculated as follows:

[0053] Score α (i,j)=(w b,i c i ) T ⊙(w e,j c j ) (13)

[0054] where w b,i ∈W b and w e,j ∈W e are learnable left and right boundary linear transformation matrices, ⊙ represents the inner product operation of the vector;

[0055] Different from the span extraction module, this module uses a character vector H as input to calculate the compatibility score between the span s and the label y∈Y, where Y = (y1, y2, ... y N ) is the label set, N is the number of labels, and the span s can be expressed as (h i ,h j ), each label in the multi-head label classifier has its own label classifier, for this label y k The label score of is calculated as follows:

[0056]

[0057] where w k is the label y k The linear transformation parameters of

[0058] This module uses the sum of span score and label score as the final entity score, and then uses softmax to get the probability that span s belongs to label y:

[0059]

[0060] In order to achieve collaborative learning between the span extraction task and the label classification task, the candidate entities obtained by the span extraction task are regarded as entity soft labels. The candidate entities and the predicted entities obtained in the current stage are used to construct the soft label constraint loss. The calculation of the soft label constraint is as follows:

[0061]

[0062] Where N is the number of labels, To predict the label of the entity, is the label of the candidate entity, and the labels of candidate entities of different categories have the same representation;

[0063] The main loss function of the method is as follows:

[0064]

[0065] Where L y is the span set of all entity labels y in the sentence, P y is the set of all non-entities or spans whose entity labels are not y in the sentence, and Score is the entity score of span s in the sentence:

[0066] Score(s,y)=(w b,i c i ) T ⊙(w e,j c j )+(wy) T [h i :h j ] (18)

[0067] Finally, the joint loss function of the method is the weighted sum of the soft label constraint loss function and the main loss function:

[0068] L=λ1L base +λ2L soft (19)

[0069] λ1 and λ2 are the weight parameters of the loss function.

[0070] Due to the adoption of the above technical solution, the technical progress achieved by the present invention is:

[0071] The present invention realizes word fusion based on a stacked grid structure, which can dynamically increase the number of layers to accommodate more word vectors that match sentences. At the same time, the structure is lossless for the transformation of word information during the storage process, and retains all the vector information and position information of the word. Combined with the hierarchical compressed sensing method, all the word information is compressed and fused layer by layer. On the one hand, this process solves the computational cost and memory requirements in the word vector fusion process, and on the other hand, it realizes more sufficient semantic fusion. This solves the problem of information loss and high computational overhead in the existing vocabulary information enhancement method based on the character-word grid structure when processing long texts.

[0072] The present invention applies a cross-attention-based fusion network to extract joint semantic features of entities at multi-granularity relationships. Multiple cross-attention networks can achieve information complementarity between sentence representation sequences at any two granularities, enriching the contextual information of entities in sentences. The method of the present invention does not only use information such as character shape and pronunciation as supplementary information of character features, but takes into account the semantic and structural relevance of entities at these different granularities.

[0073] The entity recognition method based on dual collaboration proposed in the present invention, on the one hand, overcomes the problem of insufficient accuracy and generalization ability of label classification caused by the existing span-based entity recognition method in the entity label discrimination stage, which can only use some candidate entities judged as entities in the span extraction task; on the other hand, by using a joint loss function based on soft label constraints, it realizes collaborative optimization among multiple tasks, thereby improving the accuracy and recall rate of entity recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 It is the overall flow chart of the present invention;

[0075] Figure 2 It is a diagram of the stacked grid structure of the present invention;

[0076] Figure 3 It is a schematic diagram of boundary fusion of the present invention. DETAILED DESCRIPTION

[0077] The present invention is further described in detail below in conjunction with embodiments:

[0078] In order to better describe the method proposed by the present invention, the following definitions are given here:

[0079] Definition 1 Span: In named entity recognition, the span is the character position range of the named entity in the text, usually represented as a (starting position, ending position) tuple. For a span in sentence X, it can be represented as s = (x i ,x j ).

[0080] Definition 2 Vocabulary: Extract all the words in the external vocabulary information file that needs to be introduced, remove single characters, and store the remaining words in the form of a data structure list in the vocabulary w_list.

[0081] Definition 3 Dictionary Tree: Use the vocabulary w_list to build a dictionary tree w_T. Using the vocabulary to build a dictionary tree requires traversing each word in the vocabulary, creating or visiting nodes character by character in the tree structure, and marking a leaf node at the end of the word.

[0082] Definition 4: Word vector lookup table: All words in the vocabulary and their corresponding word vectors together constitute the word vector lookup table R w .

[0083] Definition 5 Chinese character decomposition: Chinese character decomposition is to retain the independent character part of the Chinese character structure intact, and decompose the non-independent character part according to strokes.

[0084] Definition 6: Basic glyph: For a Chinese character z∈Z, where Z represents the set of all Chinese characters, the Chinese character components obtained by decomposing them in the Chinese character decomposition method are called basic glyphs.

[0085] Definition 7 Glyph set: The set of basic glyphs of all Chinese characters is called a glyph set.

[0086] Definition 8: Decomposition dictionary: The set of all Chinese characters and their corresponding glyphs is called a decomposition dictionary.

[0087] Definition 9: Glyph vector: Arrange all basic glyphs in the glyph set in ascending order of the number of occurrences to obtain a set of digital sequences, which are called glyph vectors.

[0088] Definition 10: Glyph vector lookup table: All basic glyphs in the glyph set and their corresponding glyph vectors together constitute the glyph vector lookup table.

[0089] A Chinese named entity recognition method based on stacked grid structure information enhancement includes a word fusion module based on stacked grids, a multi-granularity fusion module based on cross-attention, and an entity recognition module based on double-link collaboration. The specific method includes the following steps:

[0090] Step S1: In a word fusion module based on a stacked grid, a stacked grid structure is constructed to store all word vectors that match the characters in the sentence, and the word vector information is first compressed using hierarchical compressed sensing and then the attention mechanism is used to realize the vector fusion of characters and words;

[0091] Step S2: In the multi-granularity fusion module, multiple cross attentions are used to jointly learn the fused word vector sequence, the glyph sequence vector obtained by the glyph embedding method, and the character vector sequence to capture the relevance characteristics of the entity at multiple granularities;

[0092] Step S3: Identify the entity to be extracted and determine the category to which it belongs through the entity recognition module based on dual collaboration.

[0093] The specific steps of step S1 are as follows:

[0094] First, character embedding and word embedding are used on sentences to obtain the character vector sequence of the sentence and the word vector sequence to be matched. Then a multi-layer stacked grid structure is constructed to accommodate the word vector sequence. Finally, hierarchical compressed sensing is applied to compress the word vector information layer by layer, and multi-head attention using rotational position encoding is used to achieve vector fusion between characters and words.

[0095] Given a sentence X = (x1, x2, ..., x n ), where n is the length of the sentence, and the character vector sequence representation H of sentence X is obtained by embedding the context words:

[0096] H=(h1,...,h n )=Embedding(x1,...,x n ) (1)

[0097] Where Embedding() means using the contextual word embedding of the pre-trained model BERT, h i A vector representation of a single character.

[0098] For sentence X, first use the dictionary tree w_T to match the characters in the sentence one by one to obtain the word sequence, and then use the word vector to look up the table R w Get the corresponding word vector sequence in i s and i e Respectively represent the starting and ending positions of words in a sentence. is the vector representation of the word, and m is the number of words.

[0099] After obtaining the word vector sequence W of the sentence o Then, start building Figure 2 The stacked grid structure shown has a dimension of R L*L*L , where L is the maximum length of the sentences in the current batch of samples. The maximum number of words that this structure can accommodate is Since the structure retains the position index of the word in the original sentence sequence when storing the word vector, the structure can be restored to the original word information, so the transformation can be considered lossless. At the same time, the multi-layer stacking mode adopted enables the word vector to be aligned one by one with the character vector subsequence in the sequence, which facilitates the subsequent word vector compression process.

[0100] The stacked grid structure allows the model to accommodate more word vectors. In order to better utilize these word vector information, it is necessary to process the word vectors in combination with hierarchical compressed sensing.

[0101] In order to reduce the computational cost and memory requirements during the fusion process, the word vector needs to be compressed. The compression process consists of two parts: (1) internal compression, which compresses the word vector layer by layer and distributes the compressed word vector to all the constituent characters of the word; (2) boundary compression, which compresses the word vector information to the left and right boundaries of the word. The compressed word vector is used to enhance the first and last boundaries of the word. Then, the vectors obtained by internal compression and boundary compression are merged and fused with the character vector under the guidance of the multi-head attention mechanism. The fused word vector will contain the pairwise matching information between characters and words and the global feature information between character sequences and word sequences. This information not only enriches the contextual semantics of the entity, but also strengthens the entity boundary.

[0102] Internal compression, first initialize a unit vector sequence U = (u1, ..., u n ), and then U is combined with the word vector sequence in the stacked grid structure Perform one-by-one matching calculations:

[0103] W i F =u i W i G (2)

[0104] Get a character-by-character compressed vector

[0105] Boundary compression, such as Figure 3 As shown, the head and tail boundary matching calculations are performed on the unit vector sequence and the word vector sequence after the multi-layer grid structure is superimposed:

[0106]

[0107] Get the compressed head and tail boundary word vector W S and W E .

[0108] Then W F , W S and WE The word sequence H obtained by concatenation W The query matrix Q, key matrix K, and value matrix V for attention calculation are obtained by linear transformation with the sentence character vector sequence H:

[0109] [Q,K,V]=[Hω q ,H w ω k ,H w ω v ] (5)

[0110] where ω q ,ω k and ω v are all learnable parameter matrices.

[0111] When calculating attention, position encoding needs to be introduced to retain the position information of the input sequence. Since the characters in a sentence are more affected by adjacent information than by distal information, the rotational position encoding RoPE is used as the position encoding to achieve more attention to adjacent information. This position encoding has the advantages of simple implementation and fast calculation speed of absolute position encoding, and can also realize the ability of relative position encoding to capture the relative order relationship and relative distance between positions. The following formula is used to add position encoding to q∈Q and k∈K:

[0112] q m =f(q,m)=R f (q,m)e iΘf(q,m) =||q||e i(Θ(q)+mθ) =qe imθ (6)

[0113] k n =f(k,n)=R f (k,n)e iΘf(k,n) =||k||e i(Θ(k)+nθ) =ke inθ (7)

[0114] in The transformed vector is and Then use attention calculation to get the fused word vector W:

[0115]

[0116] where u is a learnable parameter of the attention bias.

[0117] The specific steps of step S2 are as follows:

[0118] In order to explore the relevance of information of three different granularities, namely glyphs, characters and words, in entity semantics and structure, this paper proposes a multi-granularity fusion module based on cross attention to extract the relevance features of entities at multiple granularities, and further enhance the contextual representation ability of entities. To this end, the module first uses glyph embedding to obtain the glyph vector sequence of the sentence, and then uses multiple cross attentions to perform multi-granular fusion on the obtained glyph vector sequence, the fused word vector sequence and the character vector sequence.

[0119] For sentence X, firstly, the glyph sequence of each word in the sentence is obtained by the word splitting dictionary P, and then the glyph sequence is converted into a glyph vector sequence by applying the glyph vector lookup table R. Finally, the glyph vector sequence is feature extracted using the dilated convolution DConv to obtain the glyph embedding representation G:

[0120] G=(g1,g2,...,g n )=DConv(R(P(X))) (9)

[0121] Compared with the standard convolutional neural network, the advantage of the expanded convolutional neural network is that it increases the receptive field without losing information in the pooling layer, so that each convolution input can contain a larger range of information, which is more suitable for long text sequences.

[0122] In order to capture the relevance characteristics of entities at multiple granularities, the present invention uses a fusion network based on multiple cross-attentions to obtain complementary information from word semantics and Chinese character structure. Cross-attention is widely used in machine translation and multimodal fusion. It pays attention to the semantic relationship between sequences by calculating attention on two different sequences, thus achieving information complementarity between sequences.

[0123] For sentence X, its corresponding character sequence vector is H, the fused word sequence vector is W and the glyph vector sequence is G, where the cross attention between the character vector and the word vector is calculated as follows:

[0124] Head (i) =CrossAtt(((Hω q ) (i) +u) Τ (Wω k ) (i) ,(Wω v ) (i) ) (10)

[0125] MultiHead(H,W)=[Head (1) ,Head (2) ,...,Head (l) ] (11)

[0126] where ωq ,ω k and ω v They are all learnable parameter matrices, u is the attention bias, and l is the number of heads in the multi-head cross attention.

[0127] The calculation of the cross attention between the character vector and the glyph vector is the same as the above calculation. The character-word fusion feature vector H1 and the character-glyph fusion feature vector H2 obtained by this module are input into a fully connected layer for information fusion to obtain the fusion vector C:

[0128]

[0129] in represents the connection operation, W o and b are learnable parameters.

[0130] The specific steps of step S3 are as follows:

[0131] At present, although span-based entity recognition methods can handle complex entity problems such as nested entities and overlapping entities, after the text is processed by the span extraction module, only the part recognized as an entity will be retained for subsequent entity classification. This means that after the original rich text data is screened, the number of samples used for classification will be significantly reduced, thus affecting the accuracy and generalization ability of classification. In addition, in this cascaded entity recognition method, errors in the span extraction module will directly affect subsequent entity classification, leading to error propagation.

[0132] In this regard, the present invention proposes an entity recognition method based on dual-link collaboration, and introduces entity scoring function and label scoring function in span extraction and label classification tasks respectively, which work together on the final entity recognition, and combines the joint loss function of soft label constraints to realize the collaborative processing between span extraction and label classification. And in the label classification task, the full amount of samples are used instead of the samples filtered by the span extraction module to avoid error propagation. At the same time, in order to improve the label classification effect in multi-classification tasks, the same classification strategy as that in the forward unlabeled learning model is adopted in the label classification task. By constructing a multi-head label classifier, the recognition accuracy of the method in each category is improved.

[0133] For sentence X, we need to enumerate all possible seed spans S = (s0, s1, ..., s m ), m is the number of seed spans of the sentence. Each span is represented by s=(x i ,x j ), 1≤i≤j≤n, where i and j represent the positions of the left and right boundaries of the span in the sentence, respectively, and n is the length of the sentence.

[0134] The fusion vector C obtained by the multi-granularity fusion module is sent to this module to extract candidate entities. At this time, the span vector is represented by s = (c i ,c j ), the span score is calculated as follows:

[0135] Score α (i,j)=(w b,i c i ) T ⊙(w e,j c j ) (13)

[0136] where w b,i ∈W b and w e,j ∈W e are learnable left and right boundary linear transformation matrices, and ⊙ represents the inner product operation of the vector.

[0137] Different from the span extraction module, this module uses a character vector H as input to calculate the compatibility score between the span s and the label y∈Y, where Y = (y1, y2, ... y N ) is the label set, and N is the number of labels. At this time, the span s can be expressed as (h i ,h j ), each label in the multi-head label classifier has its own label classifier, for this label y k The label score of is calculated as follows:

[0138]

[0139] where w k is the label y k The linear transformation parameters.

[0140] This module uses the sum of span score and label score as the final entity score, and then uses softmax to get the probability that span s belongs to label y:

[0141]

[0142] In order to achieve collaborative learning between the span extraction task and the label classification task, the candidate entities obtained by the span extraction task are regarded as entity soft labels. The candidate entities and the predicted entities obtained in the current stage are used to construct the soft label constraint loss. The calculation of the soft label constraint is as follows:

[0143]

[0144] Where N is the number of labels, To predict the label of the entity, is the label of the candidate entity, and the labels of candidate entities of different categories have the same representation.

[0145] The main loss function of the method is as follows:

[0146]

[0147] Where L y is the span set of all entity labels y in the sentence, P y is the set of all non-entities or spans whose entity labels are not y in the sentence, and Score is the entity score of span s in the sentence:

[0148] Score(s,y)=(w b,i c i ) T ⊙(w e,j c j )+(wy) T [h i :h j ] (18)

[0149] Finally, the joint loss function of the method is the weighted sum of the soft label constraint loss function and the main loss function:

[0150] L=λ1L base +λ2L soft (19)

[0151] λ1 and λ2 are the weight parameters of the loss function.

[0152] Evaluation of the Chinese named entity recognition method based on stacked grid structure information enhancement in this application: Experimental setup

[0153] 1 Experimental Environment and Dataset

[0154] The experimental environment of the present invention is the Linux operating system of Centos7, the processor is Intel Core i9-10900X, NVIDIA RTX A4000 graphics card, and the server has 128GB memory. The experimental framework uses the machine learning library Pytorch.

[0155] In order to verify the effectiveness of the proposed method, the present invention is evaluated on three public Chinese named entity recognition general datasets: Weibo, Resume, MSRA and two public Chinese medical Chinese named entity recognition datasets: CCKS2017 and CMeEE. Among the general datasets, the corpus of the MSRA dataset comes from news, the corpus of the Weibo dataset comes from social media, and the corpus of the Resume dataset comes from resume data from Sina Finance. Among the medical datasets, CCKS2017 is the evaluation task data of the 2017 National Knowledge Graph and Semantic Conference. The CMeEE dataset comes from the Chinese Medical Information Processing Challenge CBLUE hosted by the Alibaba Tianchi Platform. It is currently a relatively excellent public Chinese named entity recognition evaluation benchmark in the medical field in China. This dataset does not give the annotation results of the test set. The evaluation results of the test set need to be submitted to the official website of CBLUE to obtain. Detailed information about these datasets is shown in Tables 1 and 2.

[0156] 2 Evaluation indicators

[0157] The present invention uses the evaluation indicators frequently used in named entity recognition tasks: precision P (Precision), recall R (Recall) and F1 value as evaluation indicators.

[0158] Table 1 Corpus statistics of the experimental dataset

[0159]

[0160] Table 2 Statistics of named entity types in the experimental dataset

[0161]

[0162]

[0163] In order to better compare the effect of the method proposed in the present invention with the existing vocabulary enhancement method on the Chinese named entity recognition task, the YJ dictionary, which is the most commonly used in the vocabulary enhancement method, is used as the external vocabulary. At the same time, in order to verify the effect of this method in the proprietary field, the YJ dictionary and the private medical dictionary TC constructed by the Chinese pre-trained word vector of Tencent AI Laboratory are used as external vocabulary to compare with the baseline method on the Chinese medical dataset. Among them, the baseline comparison methods in the general field include Lattice LSTM, LR-CNN, LGN, PLTE, Glyce, PDMD and LWI, and the comparison baseline methods on the Chinese medical dataset include: RICON, WHSemantic+Lattice and TES-NER.

[0164] Table 3 Results of Weibo dataset (%)

[0165]

[0166] Table 4 Results of the Resume dataset (%)

[0167]

[0168] Table 5 Results of MSRA dataset (%)

[0169]

[0170]

[0171] The experimental results on the Chinese named entity recognition datasets in the general field are shown in Tables 3, 4, and 5, respectively. The contents of the table are divided into two parts: the first part is the method that does not use the pre-trained model as the character feature extractor, and the second part is the method that uses the fine-tuned pre-trained model as the character feature extractor. From the results in the table, it can be seen that on the three datasets, for the same method, the results of using the pre-trained model are better than those of not using the pre-trained model, but the improvement effect is only obvious on some models of the Weibo dataset, and the improvement effect on the Resume and MSRA datasets is average. This shows that in the Chinese named entity recognition task, the improvement brought by the pre-trained model on the recognition effect is indeed limited.

[0172] Among the comparison methods, Lattice LSTM, LR-CNN, LGN, PLTE and LWI methods only use word information, Glyce method only uses the glyph information of Chinese characters, this method uses Chinese character images to extract glyph structure information, and PDMD method uses word information and Chinese character voice features. For the Weibo dataset, the results are shown in Table 3. Compared with the best result of the method that does not use the pre-trained model, the method proposed in the present invention achieves a 14.37% improvement in F1 value. In the comparison of the methods using the pre-trained model, the F1 value is improved by 6.03% compared with the best method that only uses word information, 6.98% compared with the Glyce method, and 2.57% compared with the PDMD method. These improvement effects show that the proposed method has advantages in the introduction and use of word information and glyph information, proving the effectiveness of the word fusion module and multi-granularity fusion module in the method in feature extraction and fusion. The experimental results on the Resume dataset are shown in Table 4. Compared with the best result of the method using the pre-trained model, the F1 value is improved by 0.87%, and compared with the method without the pre-trained model, it is improved by about 2%. The experimental results on the MSRA dataset in Table 5 are similar. This shows that for the Chinese named entity recognition task, the improvement brought by the use of pre-trained models and external information is affected by the dataset to a certain extent.

[0173] Combining the results in Tables 3, 4, and 5, it can be seen that compared with other existing vocabulary enhancement methods, the proposed method shows the best results on all three datasets, among which the effect is the largest on the Weibo dataset, proving the effectiveness of the word fusion module based on the stacked grid. Compared with the method that only uses vocabulary information enhancement, the proposed method and the PDMD method introduce the Chinese character shape information and phonetic information respectively on the basis of vocabulary information, achieving better results. This shows that it is still necessary to introduce other types of external information while using the pre-trained model to improve the effect of the Chinese named entity recognition task, and also proves the effectiveness of the multi-granularity fusion module used.

[0174] Table 6 Results on CCKS2017 and CMeEE datasets (%)

[0175]

[0176] The experimental results on the dataset of professional field - medical field are shown in Table 6. The method proposed in the present invention also shows the best effect compared with the comparison method. The introduction of vocabulary information improves the precision P and recall R of the model, proving that vocabulary information is equally effective for professional field recognition. The effect comparison on the two dictionaries YJ and TC shows that although the private dictionary TC contains a large number of medical field words in its construction, it does not show an advantage over the classic YJ dictionary. This may be because the role of word information in this method is mainly to enhance entity boundary information, and the perception of the word vector representation effect is not strong.

Claims

1. A Chinese named entity recognition method based on stacked grid structure information enhancement, characterized by: The method includes a word fusion module based on stacked grids, a multi-granularity fusion module based on cross-attention, and an entity recognition module based on double-link collaboration. The specific method includes the following steps: Step S1: In a word fusion module based on a stacked grid, a stacked grid structure is constructed to store all word vectors that match the characters in the sentence, and the word vector information is first compressed using hierarchical compressed sensing and then the attention mechanism is used to realize the vector fusion of characters and words; Step S2: In a multi-granularity fusion module based on cross-attention, multiple cross-attentions are used to jointly learn the fused word vector sequence, the glyph sequence vector obtained by the glyph embedding method, and the character vector sequence to capture the correlation characteristics of the entity at multiple granularities; Step S3: Identify the entity to be extracted and determine the category to which it belongs through the entity recognition module based on dual collaboration.

2. According to claim 1, a Chinese named entity recognition method based on stacked grid structure information enhancement is characterized in that: The specific steps of step S1 are: Step S1.1 uses character embedding and word embedding on the sentence to obtain a character vector sequence of the sentence and a word vector sequence to be matched; Step S1.2 builds a multi-layer stacked grid structure to accommodate the word vector sequence; Step S1.3 applies hierarchical compressed sensing to compress the word vector information layer by layer, and uses multi-head attention using rotational position encoding to achieve vector fusion between characters and words.

3. According to claim 2, a Chinese named entity recognition method based on stacked grid structure information enhancement is characterized in that: The specific method of step S1.1 is: Given a sentence X = (x1, x2, ..., x n ), where n is the length of the sentence, and the character vector sequence representation H of sentence X is obtained by embedding the context words: H=(h1,...,h n )=Embedding(x1,...,x n ) (1) in Embedding() represents the contextual word embedding using the pre-trained model BERT, hi is the vector representation of a single character; For sentence X, use the dictionary tree for each character in the sentence w_T Perform word matching, obtain word sequences, and then use word vector lookup table Rw Get the corresponding word vector sequence in i s and i e Respectively represent the starting and ending positions of words in a sentence. is the vector representation of the word, and m is the number of words.

4. According to claim 2, a Chinese named entity recognition method based on stacked grid structure information enhancement is characterized in that: The specific method of step S1.2 is: In getting the word vector sequence of the sentence W o After that, a stacked grid structure is constructed, whose dimension size is R L*L*L , where L is the maximum length of sentences in the current batch of samples; the maximum number of words that this structure can accommodate is The multi-layer stacking mode adopted by this structure enables the word vectors to be aligned one by one with the character vector subsequences in the sequence, which facilitates the subsequent word vector compression process.

5. The Chinese named entity recognition method based on stacked grid structure information enhancement according to claim 2 is characterized in that: The specific method of step S1.3 is: The word vector is compressed. The compression process includes internal compression and boundary compression. Internal compression is to compress the word vector layer by layer and distribute the compressed word vector to all the constituent characters of the word. Boundary compression compresses the word vector information to the left and right boundaries of the word. The compressed word vector is used to enhance the first and last boundaries of the word. The vectors obtained by internal compression and boundary compression are merged and fused with the character vector under the guidance of the multi-head attention mechanism. The fused word vector contains the pairwise matching information between characters and words and the global feature information between character sequences and word sequences.

6. The Chinese named entity recognition method based on stacked grid structure information enhancement according to claim 5 is characterized by: The specific method of internal compression is: first initialize a unit vector sequence U=(u1,...,u n ), then U Sequence of word vectors in a stacked grid structure Perform one-by-one matching calculations: Get a character-by-character compressed vector The specific method of boundary compression is: perform head and tail boundary matching calculation on the unit vector sequence and the word vector sequence after superimposing the multi-layer grid structure: Get the compressed head and tail boundary word vector W S and W E ; W F , W S and W E The word sequence H obtained by concatenation W The query matrix Q, key matrix K, and value matrix V for attention calculation are obtained by linear transformation with the sentence character vector sequence H: [Q,K,V]=[Hω q ,H w oh k ,H w oh v ] (5) where ω q ,ω k and ω v are all learnable parameter matrices.

7. The Chinese named entity recognition method based on stacked grid structure information enhancement according to claim 5 is characterized by: In order to retain the position information of the input sequence during attention calculation, position encoding is introduced, and rotation position encoding RoPE is used as position encoding. The specific method of adding position encoding to q∈Q and k∈K is as follows: what m =f(q,m)=R f (q,m)e iΘf(q,m) =||q||e i(Θ(q)+mθ) =qe imθ (6) k n =f(k,n)=R f (k,n)e iΘf(k,n) =||k||e i(Θ(k)+nθ) =ke inθ (7) in The transformed vector is and Then use attention calculation to get the fused word vector W : where u is a learnable parameter of the attention bias.

8. The Chinese named entity recognition method based on stacked grid structure information enhancement according to claim 1 is characterized in that: The specific steps of step S2 are: For sentence X, firstly, the glyph sequence of each word in the sentence is obtained by the word splitting dictionary P, and then the glyph sequence is converted into a glyph vector sequence by applying the glyph vector lookup table R. Finally, the glyph vector sequence is feature extracted using the dilated convolution DConv to obtain the glyph embedding representation G: G=(g1,g2,...,g n )=DConv(R(P(X))) (9) For sentence X, its corresponding character sequence vector is H, the fused word sequence vector is W and the glyph vector sequence is G, where the cross attention between the character vector and the word vector is calculated as follows: Head (i) =CrossAtt(((Hω q ) (i) +u) Τ (Wω k ) (i) ,(Wω v ) (i) ) (10) MultiHead(H,W)=[Head (1) ,Head (2) ,...,Head (l) ] (11) where ω q ,ω k and ω v They are all learnable parameter matrices, u is the attention bias, and l is the number of heads in the multi-head cross attention; The calculation of the cross attention between the character vector and the glyph vector is the same as the above calculation. The character-word fusion feature vector H1 and the character-glyph fusion feature vector H2 obtained by this module are input into a fully connected layer for information fusion to obtain the fusion vector C: in represents the connection operation, W o and b are learnable parameters.

9. The Chinese named entity recognition method based on stacked grid structure information enhancement according to claim 1 is characterized in that: Step S3 is based on the dual-coordinated entity recognition method. The entity scoring function and the label scoring function are introduced in the span extraction and label classification tasks respectively, which work together on the final entity recognition. The joint loss function of the soft label constraint is combined to realize the collaborative processing between span extraction and label classification, and the full amount of samples are used in the label classification task. In order to improve the label classification effect in the multi-classification task, the same classification strategy as that in the forward unlabeled learning model is adopted in the label classification task. By constructing a multi-head label classifier, the recognition accuracy of the method in each category is improved.

10. The Chinese named entity recognition method based on stacked grid structure information enhancement according to claim 9, characterized in that: For sentence X, we need to enumerate all possible seed spans S = (s0, s1, ..., s m ), m is the number of seed spans of the sentence, and each span is represented by s = (x i ,x j ), 1≤i≤j≤n, where i and j represent the positions of the left and right boundaries of the span in the sentence, respectively, and n is the length of the sentence; The fusion vector C obtained by the multi-granularity fusion module is sent to this module to extract candidate entities. At this time, the span vector is represented by s = (c i ,c j ), the span score is calculated as follows: Score α (i,j)=(w b,i c i ) T ⊙(w e,j c j ) (13) where w b,i ∈W b and w e,j ∈W e are learnable left and right boundary linear transformation matrices, ⊙ represents the inner product operation of the vector; Different from the span extraction module, this module uses a character vector H as input to calculate the compatibility score between the span s and the label y∈Y, where Y = (y1, y2, ... y N ) is the label set, N is the number of labels, and the span s can be expressed as (h i ,h j ), each label in the multi-head label classifier has its own label classifier, for this label y k The label score of is calculated as follows: in wk Yes Tags yk The linear transformation parameters of This module uses the sum of span score and label score as the final entity score, and then uses softmax to get the span s Belongs to the tag y Probability: In order to achieve collaborative learning between the span extraction task and the label classification task, the candidate entities obtained by the span extraction task are regarded as entity soft labels. The candidate entities and the predicted entities obtained in the current stage are used to construct the soft label constraint loss. The calculation of the soft label constraint is as follows: Where N is the number of labels, To predict the label of the entity, is the label of the candidate entity, and the labels of candidate entities of different categories have the same representation; The main loss function of the method is as follows: Where L y is the span set of all entity labels y in the sentence, P y is the set of all non-entities or spans whose entity labels are not y in the sentence, and Score is the entity score of span s in the sentence: Score(s,y)=(w b,i c i ) T ⊙(w e,j c j )+(wy) T [h i :h j ] (18) Finally, the joint loss function of the method is the weighted sum of the soft label constraint loss function and the main loss function: L=λ1L base +λ2L soft (19) λ1 and λ2 are the weight parameters of the loss function.

Images