A Chinese nested named entity recognition method integrated with word boundary information

Abstract

The present application relates to a kind of Chinese nested named entity recognition method of fusing word boundary information, belong to the information extraction technical field in natural language processing.This method combines the language form problem specific to Chinese scene, utilizes the word level information of external vocabulary to model, for the boundary sensitive nested named entity recognition task, model pays more attention to word boundary problem, and utilizes the pre-training model to extract rich semantic information vector representation, ensure that model can better model at word level and sentence level, improve the accuracy of entity recognition, effectively solve the problem that existing machine learning Chinese nested named entity recognition model lacks word level information.

Description

Technical Field

[0001] This invention relates to a method for nested named entity recognition that incorporates lexical information. By matching text with an external dictionary, word-level information is used to provide word boundary information for nested named entity recognition, which belongs to the field of information extraction technology in natural language processing. Background Technology

[0002] As the primary tool for human communication and information transmission, natural language itself contains a wealth of unstructured information. Extracting structured data from unstructured text that is easily processed by machines is a major challenge we currently face.

[0003] Named entity recognition (NID) is a task designed to extract text fragments that can represent independent, specific transactions or entities from text. It can be used for knowledge graph construction and some downstream text analysis work.

[0004] Based on whether the boundaries of text fragments contain nested structures, named entity recognition (NENT) can be divided into flat named entity recognition and extraction and nested named entity recognition and extraction. Early research in named entity recognition generally focused on flat named entity recognition and extraction, where each word in a sentence belongs to at most one entity type, thus it was typically modeled as a sequence labeling task. In recent years, nested named entity recognition has gradually gained attention and development, employing other labeling methods to extract nested entities. However, most of this research focuses on English, with Chinese research typically involving simpler corpus transfer.

[0005] Because Chinese belongs to the Sino-Tibetan language family, unlike the Indo-European language family represented by English, English uses spaces to separate words in sentences, resulting in a relatively clear word segmentation structure. In contrast, Chinese sentences are composed of continuous characters, lacking explicit segmentation at the word level to represent basic semantic information. This leads to ambiguous word-level information in Chinese, while named entity recognition tasks are sensitive to word boundary delineation. With the widespread application of deep neural networks in natural language processing, models have improved their ability to model semantic information in sentences; however, they still lack expertise in word-level knowledge. Summary of the Invention

[0006] The purpose of this invention is to address the lack of word-level information in existing machine learning models for Chinese nested named entity recognition. This invention proposes a Chinese nested named entity recognition method that incorporates word boundary information, matching the text with an external dictionary and modeling the text together with the matched words.

[0007] First, let's explain the relevant concepts:

[0008] Definition 1: Text sequence s

[0009] This refers to a Chinese sentence containing nested entities to be extracted, consisting of consecutive characters.

[0010] The symbolic representation is: s = {w1, w2, ..., w n}, where n represents the length of the sentence, w i This represents the i-th character in the sentence.

[0011] Definition 2: External dictionary D

[0012] A dictionary file, independent of the training corpus, is a collection of common Chinese words, each word consisting of one or more characters.

[0013] The symbolic representation is: D = {d1, d2, ..., d} / D / }, where |D| represents the number of words in the dictionary file, and d i This represents the i-th word in the dictionary.

[0014] Definition 3: Text matching word M

[0015] This refers to matching a text sequence with an external dictionary using a greedy algorithm, and the set of words from the dictionary that are matched with the text is the text matching word.

[0016] The matching words corresponding to s are represented as: M={m1,…,m K}, where K represents the number of words matched in a text sequence.

[0017] Definition 4: Chinese Embedded Entity Recognition Corpus

[0018] The pointer refers to the database consisting of text and annotation information for the nested entity recognition task in the Chinese domain. The text is the set of text sequences s, and the annotation information is a triple that records the start and end indexes of the nested entity words in the original text and the entity type. It is symbolically represented as (start, end, label).

[0019] Definition 5: Chinese static word vector E

[0020] This refers to the open-source static vector representations of Chinese characters and words obtained through pre-training on a large-scale corpus. The static character vector representation is E. w Static word vectors are represented as E m .

[0021] Definition 6: Position vector PE

[0022] This refers to the vector representation corresponding to the position index pos of each character in the text sequence. The calculation method uses trigonometric functions, as shown in Equations 1 and 2:

[0023]

[0024]

[0025] Where, d model PE represents the dimension of the position vector. (pos,2i) This represents the value at the i-th dimension of the vector at index pos.

[0026] Definition 7: Pre-trained Language Model (LM) BERT

[0027] It refers to BERT, a deep language model pre-trained using large unsupervised corpora.

[0028] Definition 8: Semantic information x BERT

[0029] It refers to the sentence-level information representation obtained by calculating the information interaction between words in each sentence.

[0030] It can be achieved through pre-trained language model LM BERT Perform word-level information interaction to obtain semantic information x BERT .

[0031] Definition 9: Entity type c

[0032] The set of entity types annotated in a corpus, denoted by c = {O, c1, c2, ..., c...} |C|}, where type O represents a non-entity type, and the rest are predefined entity types.

[0033] The present invention is achieved using the following technical solution.

[0034] A method for recognizing nested Chinese named entities incorporating word boundary information includes the following steps:

[0035] Step 1: Obtain the Chinese nested entity recognition corpus, as well as external dictionaries and open-source Chinese static word vectors.

[0036] Step 2: For each text sequence s in the corpus, obtain the matching word M using a greedy algorithm.

[0037] Specifically, the greedy algorithm calculates the following:

[0038] First, iterate through each character w in the text sequence. i Search for w in the external dictionary D i The first character is w i The subsequent words can correspond one-to-one with all words of the longest possible length, and the start and end index positions of the original text corresponding to each matching word are recorded.

[0039] Then, {m1,…,m K The starting index of} is denoted as pos. M,start ={posM,s1 ,…,pos M,sK Let pos be the ending index. M,end ={pos M,e1 ,…,pos M,eK}

[0040] {w1,w2,…,w n The start and end indices of} are both incremented from 1, and are denoted as pos respectively. s,start ={1,2,…,n}、pos s,end ={1,2,…,n}.

[0041] Step 3: via pos M,start pos M,end pos s,start pos s,end Calculate the relative position vector R between every two positions. ij As shown below:

[0042]

[0043]

[0044]

[0045]

[0046]

[0047] Where, pos start,i Represents pos M,start or pos s,start The position index of the i-th element, pos end,i Represents pos M,end or pos s,end The position index of the i-th element, pos start,j pos end,j The definition is similar, and so on.

[0048] pass It reflects the boundary position information between characters and matching words. This indicates that the index is obtained using the position vector PE. The position vector, which carries the relative position encoding of boundary information; W r The parameters are the ones the model needs to learn. ReLU() is a commonly used activation function in deep learning.

[0049] Step 4: Obtain the vector representation E corresponding to the text sequence s and the matching word M from the Chinese static word vector E.

[0050] Step 5: Use a two-layer Transformer-XL encoding structure to encode the word vectors obtained in Step 4. Then, calculate the relative positions during the Attention stage to obtain a vector representation of each character with word boundary information.

[0051] Specifically, the method is as follows:

[0052] sub_layer_output=LayerNorm(x+(SubLayer(x))) (8)

[0053] Att(A,V)=softmax(A)V (9)

[0054]

[0055]

[0056] Among them, sub layer_output This represents the sub-layer output of Transformer-XL, LayerNorm() represents layer regularization calculation, SubLayer() represents the sub-layer structure of Transformer-XL, Att() represents self-attention calculation, A represents the attention weight matrix of self-attention calculation, and a ij Let A represent the element in the i-th row and j-th column of matrix A, V represent the linear transformation of the output of the previous sublayer of Transformer-XL, softmax() and ReLU() are activation functions, FFN() represents a feedforward neural network, and x i x j The characters representing the i and j index positions, x represents i x j Vector representation in Chinese static word vector E W k,E W k,R Each represents a different parameter matrix to be learned by the model, where The parameter matrices are W1, W2, W q The transpose of the matrix, b1 and b2, both represent column vectors of different parameters to be learned by the model.

[0057] Each Transformer-XL layer contains two sub-layers, and the output of the last layer is a vector representation that incorporates word boundary information, denoted as x. boundary,iEquation 8 is the basic logic of sub-layer calculation, x represents the input vector of the sub-layer, Equation 9 of the first sub-layer is the input of Equation 8, Equation 10 of the second sub-layer is the input of Equation 8, each element of the weight matrix A in Equation 9 is calculated as Equation 10, the input of the current Transformer-XL layer is the output of the previous layer, and the initial output is E.

[0058] Step 6: Using LM BERT We obtain a vector representation of each character with semantic information, and then... i The vector representation of semantic information is denoted as x. BERT,i .

[0059] Step 7: Concatenate the two vector representations according to Equation 12 to obtain the final vector representation x of the character. i .

[0060] x i =[x BERT,i ;x boundary,i (12)

[0061] Step 8: Using a biaffine classifier, calculate the span of each text segment with a start index of i and an end index of j. ij The probability p(i,j,c) for entity type c is as follows:

[0062]

[0063]

[0064]

[0065] p(i,j,c)=softmax(score ij (16)

[0066] Among them, FFNN s FFNN e This represents two independent linear mapping layers with the same dimension. Each represents a text segment obtained in step 7. ij The vector representation of the start and end characters, softmax() represents a common normalization layer, U m W m b represents the training parameter matrix of the model. m This represents the column vector of training parameters for the model.

[0067] Equations 13 to 15 describe the computation process of the dual affine classifier. First, two independent mapping layers preserve the entity start position information h in the word vector representation. start (i) Vector representation of entity end position information hend (j), after passing through a dual affine classifier, calculates the score of a text segment span(i,j) on different entity types. ij The scores are transformed into the final entity classification probability p(i,j,c) through the softmax() layer. The model selects the type with the highest probability as the entity type prediction result of span(i,j).

[0068] The model uses the cross-entropy between the model's predictions and the annotations in the corpus as the loss function for training, and continuously optimizes the loss function value to train and adjust the model parameters.

[0069] Thus far, from step 1 to step 8, word boundary information has been incorporated into the model by matching words, completing the nested named entity recognition in Chinese scenarios.

[0070] Beneficial effects

[0071] The method of the present invention has the following advantages compared with the prior art:

[0072] This method addresses the unique linguistic morphology issues in Chinese contexts by incorporating word-level information into the model using an external vocabulary. For boundary-sensitive nested named entity recognition tasks, the model focuses more on word boundary issues and utilizes pre-trained models to extract rich semantic information vector representations, ensuring that the model can better model at both the word and sentence levels, thereby improving the accuracy of entity recognition. Attached Figure Description

[0073] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0074] The method of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0075] like Figure 1 As shown, the present invention includes the following steps:

[0076] Step A: Obtain the text sequence and matching words;

[0077] Specifically, in this embodiment, matching words are obtained by matching the sequence in this article with an external dictionary, and the position information of the relevant words is recorded, which is the same as steps 1 to 2 in the invention.

[0078] Step B: Calculate the character vector representation containing word boundary information;

[0079] Specifically, in this embodiment, by sharing position information between the matching word and the text sequence, the transformer-XL structure is used to capture the word boundary information of the text corresponding to the matching word, and a character vector representation with word boundary information is obtained, which is the same as steps 3 to 5 in the invention.

[0080] Step C: Calculate the word vector representation containing semantic information;

[0081] Specifically, in this embodiment, a pre-trained model is used to calculate the word vector representation with semantic information, which is the same as step 6 in the invention.

[0082] Step D: Concatenate the two vector representations;

[0083] Specifically, in this embodiment, it is the same as step 7 in the invention description;

[0084] Step E: The dual affine classifier calculates the probability of each entity type:

[0085] In this specific embodiment, the score of each text fragment under each entity type is calculated through a double affine layer, and then converted into the probability of each entity type, which is the same as step 8 in the invention.

[0086] Example

[0087] Taking the text sequence "The Chinese delegation has won a total of 26 gold, 16 silver and 15 bronze medals" as an example, this example will explain in detail the specific operation steps of the Chinese nested named entity recognition method incorporating word boundary information described in this invention.

[0088] The processing flow of the Chinese nested named entity recognition method incorporating word boundary information is as follows: Figure 1 As shown; from Figure 1 As can be seen, the Chinese nested named entity recognition method incorporating word boundary information includes the following steps:

[0089] Step A: Obtain the text sequence and matching words;

[0090] Using a greedy algorithm to match words in a text sequence, for the text sequence "Currently, the Chinese delegation has won 26 gold, 16 silver, and 15 bronze medals" with a length of 19, dictionary matching yields the words "currently," "China," "representative," "delegation," and "won." The position of these matching words is... M,start ={1, 3, 5, 5, 9}, pos M,start ={2, 4, 6, 7, 10};

[0091] Step B: Calculate the character vector representation containing word boundary information;

[0092] Specifically, in this embodiment, Transformer-XL is used to calculate a 100-dimensional character vector representation containing word boundary information;

[0093] Step C: Calculate the word vector representation containing semantic information;

[0094] The BERT pre-trained model is used to calculate a 512-dimensional character vector representation containing semantic information for each character;

[0095] Step D: Concatenate the two vector representations;

[0096] By concatenating the two word vectors from steps B and C, a 612-dimensional word vector representation is obtained.

[0097] Step E: The dual affine classifier calculates the probability of each entity type:

[0098] Specifically, in this embodiment, a 19×19 matrix is ​​obtained through two affine layers. The elements in the matrix represent the entity type classification probability of the text segments in the text sequence with the matrix element position index as the boundary. For example, the entity probability of identifying "China" is the position of the 3rd row and 4th column in the matrix, and the entity probability of "China delegation" is the position of the 3rd row and 7th column in the matrix.

Claims

1. A Chinese nested named entity recognition method integrated with word boundary information, characterized in that, The method comprises the following steps: Step 1: Obtain a Chinese nested entity recognition corpus, and an external dictionary and an open-source Chinese static word vector; Step 2: For each text sequence s in the corpus, obtain a matching word M by using a greedy algorithm; Here, the text sequence s refers to a Chinese sentence containing nested entities to be extracted, consisting of consecutive characters, and is symbolically represented as: s = {w1, w2, ..., w...} n }, where n represents the length of the sentence, w i This represents the i-th character in the sentence; The calculation method of the greedy algorithm is as follows: First, traverse each word w in the text sequence i Find all words in the external dictionary D that start with w i and have words that follow w i that can be one-to-one corresponding as long as possible, and record the start index and end index position of the original text corresponding to each matching word pair; wherein the external dictionary D refers to a dictionary file independent of the training corpus, is a collection of Chinese common words, each word consisting of one or more characters, and the symbol is represented as: D = {d1, d2,..., d / D / }, |D| represents the number of words in the dictionary file, di represents the i-th word in the dictionary. i ​ The text matching term M refers to the set of words in the dictionary that are matched with the text sequence using a greedy algorithm. The matching terms for the text sequence s are represented as: M = {m1, ..., m} K }, where K represents the number of words matched in a text sequence; Then, the start indices of {m1,..., m K} are denoted as pos M，start = {pos M，s1 ,..., pos M，sK}, and the end indices are denoted as pos M，end = {pos M，e1 ,..., pos M，eK}; {w1, w2,..., wn}, pos n} are both from 1 and are respectively denoted as pos s，start = {1, 2,..., n}, pos s，end = {1, 2,..., n}. Step 3: Calculate the relative position vector R between each two positions by starting index pos M，start , ending index pos M，end , pos s，start , pos s，end ij As follows:​ wherein pos start，i represents the position index of the i-th in pos M，start or pos s，start wherein pos end，i represents the position index of the i-th in pos M，end or pos s，end wherein pos start，j , pos end，j are defined similarly, and so on. By embodies the boundary position information between the word, the matching word, indicates that the index is obtained by using the position vector PE The position vector carries the relative position code of the boundary information; W r is the parameter that the model needs to learn, and ReLU() is a commonly used activation function in deep learning; The position vector PE refers to a vector representation corresponding to the position index pos of each word in the text sequence, and is calculated by using a trigonometric function, as shown in formula 6 and formula 7: wherein d model denotes the dimension of the position vector, PE (pos，2i) denotes the value at the i-th dimension of the position vector of the pos-th index. Step 4: Obtain the vector representation E corresponding to the text sequence s and the matching word M from the Chinese static word vector E, The Chinese static word vector E refers to an open-source static vector representation of Chinese characters and words obtained by pre-training based on a large corpus, wherein the static word vector representation is E w , and the static word vector representation is E m ; Step 5: Using the encoding structure of two-layer Transformer-XL, the word vector obtained in step 4 is encoded and calculated, and the relative position in the Attention stage is fused to obtain the vector representation of each word with word boundary information; Specifically, the method is as follows: sub_layer_output=LayerNorm(x+(SubLayer(x))) (8) Att(A,V)=softmax(A)V (9) wherein, sub layer_output represents the output of the sub-layer of the Transformer-XL, LayerNorm() represents the layer normalization calculation, SubLayer() represents the sub-layer structure of the Transformer-XL, Att() represents the self-attention calculation, A represents the attention weight matrix of the self-attention calculation, a ij represents the element in the i-th row and the j-th column of the A matrix, V represents the result of the linear transformation of the output of the previous sub-layer of the Transformer-XL, softmax(), ReLU() are activation functions, FFN() represents a feedforward neural network, x i , x j represents the character at the i, j index position, represents x i , x j represents the vector in the Chinese static word vector E, W k，E , W k，R all represent different parameter matrices to be learned by the model, wherein is the transpose matrix of the parameter matrix W1, W2, W q , b1, b2 all represent different parameter column vectors to be learned by the model; In each Transformer-XL layer, there are two sub-layers, and the output of the last layer is the vector representation of the fusion word boundary information, denoted as x boundary，i , formula 8 is the basic logic of sub-layer calculation, x represents the input vector of the sub-layer, the first sub-layer formula 9 is the input of formula 8, the second sub-layer formula 10 is the input of formula 8, and each element of the weight matrix A in formula 9 is calculated as formula 10, the input of the current Transformer-XL layer is the output of the previous layer, and the initialized output is E; Step 6: Utilizing a pre-trained language model LM BERT to obtain a vector representation of each word w i with semantic information BERT，i ; Semantic information refers to information representation at the sentence level obtained by calculating the information interaction between words in each sentence; Step 7: Concatenate the two vector representations according to formula 12 to get the final word vector representation x i ; x i = [x BERT，i ; x boundary，i ] (12) Step 8: Compute, using the bi-affine classifier, the probability p(i,j,c) for each text span starting at index position i and ending at index position j ij For the probability p(i,j,c) for entity type c: p(i, j, c) = softmax(score ij ) (16) where FFNN s , FFNN e denotes two independent linear mapping layers of the same dimension, denote the vector representation of the start character and the end character of the text span span ij obtained from step 7, respectively, and softmax() denotes the common normalization layer, U m , W m denote the training parameter matrix of the model, and b m denote the training parameter column vector of the model; The entity type c refers to a set of entity types labeled in the corpus, and is denoted by c = {0, c1, c2,..., c |C|}, where 0 type represents a non-entity type, and the rest are predefined entity types; Equations 13-15 are the calculation process of the bi-affine classifier; first, the word vector representation with entity start position information h is retained through two independent mapping layers start (i), vector representation of entity end position information h end (j), the score score of the text segment span(i, j) on different entity types is calculated through the bi-affine classifier ij The score is converted into the final entity classification probability p(i, j, c) through the softmax() layer, and the model selects the type with the maximum probability as the entity type prediction result of span(i, j). The cross entropy of the prediction result of the model and the labeled result in the corpus is used as the loss function of the model training, and the model parameters are trained and adjusted by continuously optimizing the loss function value.

Images