Chinese address named entity recognition method and device based on word information fusion

Abstract

The application discloses a Chinese address named entity recognition method and device based on word information fusion. The method mainly includes three parts: a word information generation network construction, a label distribution learning network construction and a character label learning network construction. The application aims to obtain and integrate word information in text representation. The word information is represented by n-gram fragments. In view of the defect that the character model lacks sufficient context information, the application integrates character information and word information. The word information generation source is selected based on the character word position information learned by the label learning network. The more suitable n-gram fragments representing the word to which the character belongs are selected by the word position information. Finally, the model is given sufficient and correct context information to learn the full text information and carry out the downstream named entity recognition task. The application ensures that the integrated word information is based on the original data, accelerates the speed of the model to obtain specific words, and improves the accuracy of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of Chinese address named entity recognition, and in particular to a Chinese address named entity recognition method and device based on word information fusion. BACKGROUND

[0002] With the rapid development of informatization, the fields of take-out, postal and financial risk control, which are highly related to addresses, are also stepping onto the digitalization track. Chinese address named entity recognition refers to identifying various address-related entities from text, and based on the entities, subsequent related work is carried out. The efficiency of entity recognition will greatly affect the subsequent tasks. In particular in the Chinese field, due to the lack of obvious separation like spaces in the English field, a single character does not have special semantic information. In the task of Chinese named entity recognition, the first priority is to correctly segment a Chinese sentence. It is very difficult to do this without human prior knowledge. For example, in the sentence "She said it is reasonable", "it is" and "it is reasonable" are also correct words from the perspective of word segmentation, but in the actual context, based on human prior knowledge, the sentence should be segmented as "she / said / it is / reasonable". Named entity recognition also needs to identify words and classify them according to the context and attributes of the words.

[0003] Currently, there are three main ways to integrate additional information into character vectors in Chinese named entity recognition: 1) searching for words ending with the current character in the word table, and inputting all the found words as additional information into the model together with the character; 2) searching for word vectors containing the current character in the word table and integrating them according to certain rules, and then inputting the obtained word vectors into the model; 3) integrating the label probabilities of the current character in all data, and then inputting the probability vector into the model.

[0004] The first method needs to search for words for each character, so the number of word units needed to be added to each data is different, which causes the model to be unable to be trained in batches, resulting in the problem of slow training speed. The second method searches for words in the word table, but the found words may not conform to the text character words, and may introduce incorrect word noise. The third method simply adds label information to character information, lacking the most critical word information.

[0005] In summary, the traditional time series data query method cannot meet the following requirements at the same time:

[0006] 1) quickly obtain word information, and integrate word, character and label probability information into the input character information.

[0007] 2) In order to improve the accuracy of model named entity recognition while providing more information for the model, the efficiency of model training and online prediction is ensured, and a good balance between model precision and speed is achieved. SUMMARY

[0008] In view of the problems of the traditional Chinese named entity recognition method, the application provides a Chinese address named entity recognition method and device based on word information fusion, which effectively introduces word information into the model from text data in the field of Chinese address, and comprehensively recognizes named entities based on character information and word information. The generation source of the word information is selected as an n-gram segment representation based on character word position information, so as to provide sufficient information for the model to perform the named entity recognition task.

[0009] The purpose of the application is achieved by the following technical scheme: in the first aspect, the application provides a Chinese address named entity recognition method based on word information fusion, which comprises the following steps:

[0010] (1) obtaining the n-gram segment vector representation X=(x1, x2,...,x n ) of the Chinese address, and obtaining the corresponding real word segment Y=(y1, y2,...,y m ); wherein n is the number of characters in the n-gram segment, and m is the number of characters in the real word segment;

[0011] (2) constructing a word information generation network, adopting a double-tower model structure, and the network specifically operates as follows:

[0012] (2.1) inputting the n-gram segment and the real word segment into the word information generation network, and obtaining random character vector coding through the Embedding layer;

[0013] (2.2) learning the character vector representation through the ELMO layer and the Dense layer after the character vector coding;

[0014] (2.3) after the character vector representation is subjected to the mean pooling layer (meanpooling), the text segment representation is a word vector;

[0015] (2.4) in the classification learner, the word vector of the n-gram segment and the word vector of the real word segment are spliced, then the difference value and the dot product between the two word vectors are spliced, so as to obtain the relationship feature between the words, and after the full connection layer, the vector dimension is mapped to a two-dimensional space, and the similarity between the two vectors is judged;

[0016] (3) constructing a word information acquisition network, which comprises a label distribution learning network and a character label learning network;

[0017] The label distribution learning network obtains the character vector representation of the n-gram segment in the same way as the vocabulary information generation network, extracts a text feature code, uses a fully connected layer as a decoder, and obtains a probability distribution P of the vocabulary corresponding label label As a state matrix of a conditional random field, label inference is performed by the conditional random field;

[0018] The character label learning network specifically operates as follows:

[0019] (3.1) Select the character vector E output by the Embedding layer of the label distribution learning network C As part of the output of the embedding layer;

[0020] (3.2) According to the different positions of the current word in the n-gram and the word position annotation category q, obtain the word vector set before the last Dense layer through the vocabulary information generation network The word vector for the annotation category q;

[0021] (3.3) According to the probability distribution P of the label obtained by the label distribution learning network label, Learn the probability P of each character label belonging to each word position annotation pos ;

[0022] (3.4) According to the word vector set E τ obtained in step (3.2) and the word position annotation probability T pos obtained in step (3.3), obtain the vocabulary information E W in the embedding layer through tensor product ;

[0023] (3.5) Combine the character vector E C and the vocabulary information E W in the embedding layer, input into the WP-LSTM model, and then use the Dense layer and the conditional random field as the decoder and the label inference layer to output Z = (z1, z2,..., z n ) as the predicted label, finally learn the character relationship in Chinese address named entity recognition, and realize Chinese address named entity recognition.

[0024] Further, ELMO is a network structure composed of two bidirectional LSTMs (Bidirectional LSTM); the final vector representation of the ELMO layer is:

[0025]

[0026] Wherein is the character vector of the i-th position, γtask is a coefficient related to the pre-training task, and L is the number of layers, is the weight coefficient of the normalized relevant layer, is the output vector of the jth layer BiLSTM, contains the previous information, contains the subsequent information.

[0027] Further, in the training process, the loss of ELMO synthesizing the previous and subsequent information is the training target, that is, the optimization of the following loss:

[0028]

[0029] where θ x represents the character input vector, represents the forward LSTM parameter, represents the reverse LSTM parameter, θ S represents the softmax layer, p represents the probability, t k represents the position k of the text.

[0030] Further, the text segment can be represented as a word vector, specifically:

[0031] E vector = mean(sum(H vector ))

[0032] where H vector represents the output vector of the previous layer, vector is X or Y, that is, X and Y are encoded into word vector features E X and E Y of the same dimension by the formula.

[0033] Further, the splicing operation of step (2.4) is as follows:

[0034] E = [E X , E Y , E X -E Y , E X ⊙E Y ]

[0035] where E is the spliced vector.

[0036] Further, the conditional random field learns the transition probability between labels through the label probability distribution Pl abel and the real label Pg old , and infers the probability of all labels through the following formula:

[0037]

[0038] where O = WoH + bo represents the current character label probability, Wo, b o respectively represent the parameter matrix and the parameter vector, and H is the output of the previous layer. is the trainable parameter for a label pair (y i-1 , y i ), represents the transition probability between them, and p(y|s) represents the probability of label y under the condition of weight s; the goal of the conditional random field is to obtain the label order y * with the maximum score under the given text input condition.

[0039] Further, in step (3.2), for the word position annotation appearing multiple times, a word vector is selected from all corresponding positions with equal probability as the word vector of the word position annotation.

[0040] In a second aspect, the present application provides a Chinese address named entity recognition device based on word and character information fusion, comprising a memory and one or more processors, the memory stores executable code, and the processor executes the executable code to realize the Chinese address named entity recognition method based on word and character information fusion.

[0041] In a third aspect, the present application provides a computer readable storage medium, which stores a program, and the program is executed by a processor to realize the Chinese address named entity recognition method based on word and character information fusion.

[0042] The present application has the following technical features:

[0043] 1. Fewer noise and faster search speed: in the vocabulary information generation stage, each n-gram fragment containing a specific character and the vocabulary to which it belongs are input into the vocabulary information generation network, so that the n-gram fragment and the corresponding real vocabulary have similar word vector representations, and the association can be learned. Through this network, the model can obtain the vocabulary vector corresponding to each character in the training data in batches, and the obtained vocabulary vector is based on the internal real vocabulary of the text, and does not need to rely on an additional external large vocabulary for search.

[0044] 2. The encoding information is rich: in the vocabulary information acquisition stage, the vocabulary vector is integrated into the vocabulary vector based on the word position distribution probability and the corresponding n-gram segment. During the training process, the model first predicts the output classification soft label and the character vector through a label distribution learning network, and trains the network classification accuracy through the real label; then the vocabulary information acquisition network obtains the word position distribution probability of the character according to the soft label, generates the vocabulary vector based on the word position distribution probability and the vocabulary information, and predicts the final network classification after combining the character vector. This network can not only combine the character itself vocabulary vector, but also learn the weight of different vocabulary vectors through the character itself word position label distribution, finally obtain the vocabulary information, and provide more rich segmentation and labeling knowledge for the Chinese named entity recognition task. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0046] Figure 1 A Chinese address named entity recognition method with word information fusion provided by the present application;

[0047] Figure 2 A vocabulary information generation network structure schematic diagram;

[0048] Figure 3 A vocabulary information acquisition network structure schematic diagram;

[0049] Figure 4 A WP-LSTM model structure schematic diagram;

[0050] Figure 5 A Chinese address named entity recognition device with word information fusion provided by the present application. DETAILED DESCRIPTION

[0051] The specific embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0052] The present application provides a method of integrating character and vocabulary information in Chinese text to assist accurate named entity recognition, obtaining the word vector of the word based on the continuous character segment and the character label probability, giving the model a large amount of prior knowledge, and improving the accuracy of the model for address text named entity recognition. The present application is mainly suitable for the technical fields of financial risk control, user marketing, etc.

[0053] The Chinese address named entity recognition method based on word information fusion provided by this invention consists of two parts: the first part is a word information generation network for generating word information, and the second part is a word information acquisition network for acquiring word information. The relationship between the two parts is as follows: Figure 1 As shown:

[0054] In the vocabulary information generation network, the training objective is to learn the similarity between n-gram segments and real word vectors, employing a dual-tower model structure. The specific structure is as follows: Figure 2 As shown:

[0055] In this network, the input consists of two parts, represented by an n-gram segment vector as x = (x1, x2, ..., x...). n ), where n is the number of characters in the segment, and the real word vector is represented as Y = (y1, y2, ..., y3). m ), where m is the number of characters in the actual vocabulary. The learning steps are as follows:

[0056] 1) Obtain random character vector encoding through the Embedding layer.

[0057] 2) Character vector representations are learned through ELMO and Dense layers. ELMO is a network structure composed of two bidirectional LSTMs. The final vector representation of the ELMO layer is as follows:

[0058]

[0059] in Let be the character vector at position i, γtask be the coefficients related to the pre-training task, and L be the number of layers. These are the weight coefficients of the relevant layers after normalization. Let j be the output vector of the j-th BiLSTM layer. Including the information above, Includes information below.

[0060] During training, ELMO uses the loss from both the preceding and following context as the training objective, i.e., optimizing the following: oss :

[0061]

[0062] Where θ x Represents a character input vector. Indicates the parameters of the forward LSTM. θ represents the parameters of the inverse LSTM. s This represents a softmax layer, where p represents probability and t represents the probability. k This represents the text at position k.

[0063] 3) After mean pooling, the text segment can be represented as a word vector, i.e.:

[0064] E vector = mean(sum(H vector ))

[0065] where H vector represents the output vector of the previous layer, and vector is X or Y, i.e. x and Y are encoded into word vector features E X and E Y by the formula.

[0066] 4) In the classification learner, the difference and dot product between the two word vectors of the concatenated n-gram segment and the real word segment are concatenated to obtain the inter-word relationship feature, i.e.:

[0067] E = [E X , E Y , E X -E Y , E X ⊙E Y ]

[0068] where E is the concatenated vector.

[0069] The final vector constructed in this way contains direct and indirect features between words, and the similarity between the two vectors is determined.

[0070] After the final full connection layer, the vector dimension is mapped to a two-dimensional space, and the parameter update is based on the cross-entropy loss of binary classification. The finally trained classification learner can accurately judge the similarity of two word segments, and it can also be considered that the word vector result taken out in the encoder is also similar. Based on this word information generation network, the n-gram word vector can also have similar performance as the real word vector in subsequent tasks, without searching for real words in the word table.

[0071] In the word information acquisition network, the main learning is the integration of word information generation, including (a) part--label distribution learning network and (b) part--character label learning network, as shown in the structure: Figure 3

[0072] In the word information acquisition network (a) part--label distribution learning network, the learning steps are as follows:

[0073] 1) After obtaining the character vector encoding, pass through the ELMO layer and the Dense layer, and use BiLSTM as a text feature encoder to extract text feature encoding.

[0074] ​2) Use a fully connected layer as a decoder to obtain the probability distribution P of the labels. label The state matrix of a Conditional Random Field (CRF) is then used for label inference, learning the label transition probabilities in its transition matrix. The CRF uses the label probability distribution P... label and real label P gold Learn the transition probabilities between labels, and infer the probabilities of all labels using the following formula:

[0075]

[0076] in O represents all possible label orders for the text input C, where O = WoH + bo represents the probability of the current character label, and Wo and b represent the probability of the current character label. o These represent the parameter matrix and parameter vector, respectively, with H being the output of the previous layer. It refers to a label pair (y i-1 y i The trainable parameters of the conditional random field (CRF) represent the transition probabilities between them, and p(y|s) represents the probability of label y given weights s. The goal of CRF is to obtain the label order y that maximizes the score given a text input C. * .

[0077] In the vocabulary information acquisition network (b) – character label learning network, the learning steps are as follows:

[0078] 1) Select the character vector E output by the label distribution learning network after the embedding layer. C As part of the output of the embedding layer.

[0079] 2) Based on the different positions of the current character in the n-gram and the word position annotation type q, obtain the set of word vectors before the last Dense layer through the vocabulary information generation network. Let q be the word vector of the tagging category q; where q is the tagging category. For a tagging category that appears multiple times, select one word vector from all corresponding positions with equal probability as the word vector of this tagging category.

[0080] 3) Learn the probability distribution P of the labels from the label distribution learning network. label, Learn the probability P of each character tag belonging to each word position tag. pos .

[0081] 4) Based on the word vector set E obtained in step 2), τ And the word position labeling probability P obtained in step 3) pos Through tensor product Obtain lexical information E from the embedding layer W .

[0082] 5) Combine character vector E C and word information E in embedding layer w , i.e. E = [ E C, E w ] as input of WP-LSTM. Figure 4 The WP-LSTM model structure is shown in Figure 4 , where <pad> is a character padding symbol, which is a randomly initialized fixed vector, and when the position of the n-gram segment exceeds the length of the text itself, the character pointed to c in the figure, such as "Zhe", "Jiang", and "Sheng", is encoded as E c , the segment pointing to the w part is encoded as E w , and finally fused together.

[0083] 6) Use Dense layer and conditional random field as decoder and label inference layer, output Z = (z1, z2,..., z n ) as predicted label, finally learn character relationship in Chinese address named entity recognition, and realize Chinese address named entity recognition.

[0084] Corresponding to the above-mentioned embodiment of the Chinese address named entity recognition method fused with word information, the present application also provides an embodiment of a Chinese address named entity recognition device fused with word information.

[0085] Referring to Figure 5 , the Chinese address named entity recognition device fused with word information provided by the embodiment of the present application comprises a memory and one or more processors, the memory stores executable code, and the processor executes the executable code to implement the Chinese address named entity recognition method fused with word information in the above-mentioned embodiment.

[0086] The embodiment of the Chinese address named entity recognition device fused with word information of the present application can be applied to any device with data processing capability, which can be a device or apparatus such as a computer. The device embodiment can be realized by software, or by hardware or a combination of software and hardware. Taking software realization as an example, as a logical device, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory and running by the processor of the device with data processing capability where it is located. From the hardware level, as shown in Figure 5 , it is a hardware structure diagram of the device with data processing capability where the Chinese address named entity recognition device fused with word information of the present application is located. In addition to the Figure 5In addition to the processor, the memory, the network interface, and the nonvolatile memory, any data processing capable device in which the apparatus of the embodiments is located can also include other hardware according to the actual functions of the data processing capable device, which will not be described herein.

[0087] The implementation processes of the functions and roles of the units in the apparatus are specifically described in the implementation processes of the corresponding steps in the above method, which will not be described herein.

[0088] For the apparatus embodiments, since they basically correspond to the method embodiments, the related parts can be referred to the part of the method embodiments. The apparatus embodiments described above are merely illustrative, and the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place or distributed on multiple network units. Some or all of the modules can be selected to achieve the purpose of the present application according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0089] The embodiments of the present application also provide a computer readable storage medium, which stores a program, and the program is executed by a processor to implement the word information fusion Chinese address naming entity recognition method in the above embodiments.

[0090] The computer readable storage medium can be an internal storage unit of any data processing capable device, such as a hard disk or a memory. The computer readable storage medium can also be an external storage device of any data processing capable device, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. Further, the computer readable storage medium can include both the internal storage unit and the external storage device of any data processing capable device. The computer readable storage medium is used to store the computer program and other programs and data required by the data processing capable device, and can also be used to temporarily store data that has been output or will be output.

[0091] The above embodiments are used to explain and describe the present application, but not to limit the present application. Any modifications and changes made to the present application within the spirit and protection scope of the claims of the present application all fall within the protection scope of the present application.

Claims

1. A method for Chinese address named entity recognition based on word and phrase information fusion, characterized in that, The method includes the following steps: (1) Obtain the n-gram fragment vector representation of the Chinese address as X=(x1,x2,…,x n ), and obtain the corresponding real word fragments Y = (y1, y2, ..., y m ); where n is the number of characters in the n-gram segment, and m is the number of characters in the actual vocabulary segment; (2) Construct a vocabulary information generation network using a dual-tower model structure. The specific operation of this network is as follows: (2.1) Input n-gram fragments and real vocabulary fragments into the vocabulary information generation network, and obtain random character vector encodings through the Embedding layer; (2.2) The character vector encoding is learned through the ELMO layer and the Dense layer; (2.3) Character vector representation: After mean pooling, the text segment is represented as a word vector; (2.4) In the classification learner, the word vectors of the n-gram segment and the word vectors of the real vocabulary segment are concatenated, and then the difference and dot product between the two word vectors are concatenated to obtain the word relationship features. After passing through the fully connected layer, the vector dimension is mapped to a two-dimensional space to determine the similarity between the two vectors. (3) Construct a vocabulary information acquisition network, which includes two parts: a label distribution learning network and a character label learning network; The label distribution learning network obtains the character vector representation of n-gram segments in the same way as the vocabulary information generation network, extracts text feature encoding, uses fully connected layers as decoders, and obtains the probability distribution P of the labels corresponding to the words. label As the state matrix of a conditional random field, label inference is performed using the conditional random field. The specific operation of the character label learning network is as follows: (3.1) Select the character vector E output by the label distribution learning network after the embedding layer. C As part of the output of the embedding layer; (3.2) Based on the different positions of the current character in the n-gram and the word position annotation type q, obtain the set of word vectors before the last Dense layer through the vocabulary information generation network. The word vectors labeled with category q; (3.3) The probability distribution P of the labels obtained by learning the network based on the label distribution. label Learn the probability P of each character tag belonging to each word position tag. pos ; (3.4) Based on the word vector set E obtained in step (3.2) τ The word position labeling probability P obtained in step (3.3) pos Through tensor product Obtain lexical information E from the embedding layer W ; (3.5) Combining character vector E C and lexical information E in the embedding layer W Input a WP-LSTM model, then use a Dense layer and a Conditional Random Field as the decoder and label inference layer, and output Z = (z1, z2, ..., z n The system uses the predicted labels to learn the character relationships in Chinese address named entity recognition, thereby achieving Chinese address named entity recognition.

2. The Chinese address named entity recognition method based on word and phrase information fusion according to claim 1, characterized in that, ELMO is a network structure consisting of two bidirectional LSTMs; the final vector representation of an ELMO layer is as follows: in Let γ be the character vector at position i. task The coefficients are related to the pre-training task, where L is the number of layers. These are the weight coefficients of the relevant layers after normalization. Let j be the output vector of the j-th BiLSTM layer. Including the information above, Includes information below.

3. The Chinese address named entity recognition method based on word and phrase information fusion according to claim 2, characterized in that, During training, ELMO uses the loss from both the context and the training context as the training objective, that is, to optimize the following loss: Where θ x Represents a character input vector. Indicates the parameters of the forward LSTM. θ represents the parameters of the inverse LSTM. s This represents a softmax layer, where p represents probability and t represents the probability. k This represents the text at position k.

4. The Chinese address named entity recognition method based on word and phrase information fusion according to claim 1, characterized in that, Text fragments can be represented as word vectors, specifically: E vector =mean(sum(H vector )) Among them, H vector This represents the output vector of the previous layer, where vector is either X or Y. Specifically, X and Y are encoded into word vector features of the same dimension using a formula. X and E Y .

5. The Chinese address named entity recognition method based on word and phrase information fusion according to claim 1, characterized in that, The splicing operation in step (2.4) is as follows: And=[And X ,AND Y ,AND X -AND Y ,AND X ⊙E Y ] Where E is the concatenated vector.

6. The Chinese address named entity recognition method based on word and phrase information fusion according to claim 1, characterized in that, Conditional random fields use label probability distribution P label and real label P gold Learn the transition probabilities between labels, and infer the probabilities of all labels using the following formula: in Represents all possible label orders for the text input, O = W o H+b o W represents the current character label probability. o b o These represent the parameter matrix and parameter vector, respectively, with H being the output of the previous layer. It refers to a label pair (y i-1 ,y i The trainable parameters of ) represent the transition probabilities between them, and p(y|s) represents the probability of label y given weight s; the goal of the Conditional Random Field is to obtain the label order y that maximizes the score given the text input. * .

7. The Chinese address named entity recognition method based on word and phrase information fusion according to claim 1, characterized in that, In step (3.2), for word position tags that appear multiple times, a word vector is selected from all corresponding positions with equal probability as the word vector for this word position tag.

8. A Chinese address named entity recognition device that integrates word and phrase information, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements a Chinese address named entity recognition method based on word information fusion as described in any one of claims 1-7.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements a Chinese address named entity recognition method based on word information fusion as described in any one of claims 1-7.

Images