A Chinese named entity recognition method based on selective fusion of multimodal information

Abstract

This invention relates to a Chinese named entity recognition method based on selective fusion of multimodal information, belonging to the field of natural language processing technology. This invention effectively solves the problem of how to effectively integrate the two important pieces of information—phonetic and glyphic features—into named entity recognition. By adding the pinyin and radical sequences of characters carrying semantic information to the input of named entity recognition, the vectors are endowed with stronger semantic information. The selective fusion method can dynamically control the weight of phonetic and glyphic features, effectively improving the performance of named entity recognition. This invention can provide effective support for natural language processing tasks such as machine translation, question answering systems, and reading comprehension.

Description

Technical Field

[0001] This invention relates to a Chinese named entity recognition method, specifically a Chinese named entity recognition method that selectively integrates multimodal information such as vocabulary, pronunciation, and character shape, and belongs to the field of natural language processing technology. Background Technology

[0002] Named Entity Recognition (NER) technology aims to detect the boundaries of entity references and determine their categories from a given text sequence. This technology forms the basis for many downstream tasks in natural language processing, such as machine translation, question answering systems, and reading comprehension. The main challenge in NER research lies in the semantic analysis of Chinese text, manifested in difficulties in lexical information fusion and the incorporation of semantic information from Chinese characters.

[0003] Unlike English text, Chinese words do not have spaces as boundaries, and the semantics that a single character can represent are limited. Therefore, how to integrate lexical information into the model has become a research hotspot in many Chinese natural language processing tasks.

[0004] In the task of Chinese named entity recognition, in order to make full use of the richer semantic information of vocabulary, researchers have designed neural network models based on various resources such as automatic word segmenters, multi-character representations, and external dictionaries. Among them, the word segmentation-entity recognition framework, which operates in a pipeline manner, is limited by the accuracy of Chinese word segmentation and performs poorly in low-resource domains. While multi-character representations contain co-occurrence information between characters and supplement contextual information to some extent, most multi-character combinations do not have actual word meaning and cannot well simulate vocabulary information.

[0005] To avoid the error propagation problem of word segmenters and the semantic deficiencies of multi-character representations, combining external dictionaries with character-level NER models has become the mainstream word information fusion approach. Among them, Lattice-LSTM injects all potential matching word information from the input text into the single-character representation, achieving performance improvements on Chinese NER datasets across multiple domains. This research has also spurred a surge in research on vocabulary information fusion. Specifically, PLTE and FLAT, based on the Transformer framework, have designed two word information fusion methods. PLTE, based on a transformer encoder, can process the modeling of all characters and matching dictionary word information in parallel in batches. Furthermore, it adds positional relation representations, introduces a porous mechanism to enhance locality modeling and maintain the ability to capture long-term dependencies, endowing character vectors with richer semantic information, and achieving performance improvements on multiple datasets. FLAT concatenates all potential words matching the current input text, expanding the original single-character sequence into a word sequence. This flattens the grid structure into spans, and then, based on the Transformer's self-attention mechanism and the relative position encoding of the spans, allows each character to directly interact with potential matching words, thus improving the performance of the NER model. MECT, building upon FLAT, takes into account the pictographic nature of Chinese characters by incorporating radicals (components) with semantic information into the character vectors. It also uses a Cross-Transformer network module to interactively fuse lattice embeddings and radical embeddings to enhance semantic information.

[0006] Despite the various efforts mentioned above to incorporate lexical information and enhance semantic information, the pronunciation and form of Chinese characters are two crucial pieces of information, carrying significant syntactic and semantic information in language understanding tasks. To date, no method has fully integrated these two important pieces of information into NER methods. Summary of the Invention

[0007] The purpose of this invention is to address the common problem in current Named Entity Recognition (NER) methods that neglect the two crucial pieces of information in Chinese characters: pronunciation and form. To solve the technical challenge of effectively integrating these two important pieces of information into NER methods, this invention creatively proposes a multimodal information selective fusion method for Chinese Named Entity Recognition. This method can better dynamically fuse the semantic information among characters, pronunciation, and form.

[0008] The innovation of this invention lies in incorporating the pinyin and radical sequences of characters carrying semantic information into the input of NER (Named Entity Recognition). A CNN (Convolutional Neural Network) is used to encode the pinyin and radical of Chinese characters. After interactive embedding through a Cross-Transformer network, selective fusion is employed to dynamically generate fusion weights for different modalities.

[0009] First, the semantic (Lattice) information of words in the original text is obtained using a dictionary matching method, and relative position encoding derived from the beginning and end positions is used to adapt the word boundary information.

[0010] Then, for each character in the text, a CNN network is used to extract the pinyin and radical information of the Chinese character. After obtaining the pinyin, radical, and lattice information of the Chinese character, a CrossTransformer is used to enhance the semantic information between lattice and pinyin, and between lattice and radical.

[0011] Then, these vectors are integrated using a selective fusion method.

[0012] Finally, the word portion is masked, and the fused information is passed to a conditional random field to obtain the final label prediction probability, thus completing Chinese named entity recognition.

[0013] Beneficial effects

[0014] Compared with the prior art, the present invention has the following advantages:

[0015] 1. This method, based on lexical information, adds phonetic and glyphic information carrying important semantic information, giving the vector stronger semantic information. Furthermore, the designed selective fusion module can dynamically integrate information from various modalities, effectively improving the performance of named entity recognition.

[0016] 2. This invention employs a selective fusion method, which dynamically controls the weighting of phonetic and glyphic information, effectively improving named entity recognition performance. It provides effective support for natural language tasks such as machine translation, question answering systems, and reading comprehension. Attached Figure Description

[0017] Figure 1 This is a diagram of the overall architecture of the method of this invention;

[0018] Figure 2This is a diagram of the model Lattice Embedding construction for step 1 of the method of the present invention and its embodiment;

[0019] Figure 3 This is a construction diagram of the model Pinyin and Radical Embedding for step 1 of the method of the present invention and an embodiment;

[0020] Figure 4 This is a Cross-Transformer network diagram of step 2 of the method of the present invention and its embodiments;

[0021] Figure 5 This is the selective fusion layer in step 3 of the method of the present invention and in the embodiments. Detailed Implementation

[0022] The method of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0023] The objective of this invention is achieved through the following technical solution:

[0024] A method for Chinese named entity recognition that selectively fuses multimodal information includes the following steps:

[0025] Step 1: Input multimodal information.

[0026] Obtain semantic (Lattice) information, phonetic (Pinyin) information, and radical (Radical) information of Chinese characters.

[0027] Step 1-1: Semantic information acquisition.

[0028] First, the semantic information of Chinese characters is obtained using dictionary matching methods, and relative position encoding derived from the beginning and end positions is used to adapt to word boundary information.

[0029] Then, Lattice Embedding is obtained by initializing with pre-trained word vectors.

[0030] Step 1-2: Obtaining phonetic information.

[0031] To obtain the pronunciation of Chinese characters (for example, you can use the Python library tool pypinyin), including the initial consonant, final vowel, and tone.

[0032] The pronunciation of each Chinese character is combined in the order of initial consonant, final vowel, and tone, and then input into a CNN convolutional network for embedding representation according to Equation 1:

[0033] x i =f(w·ei:i+h-1 +b) (1)

[0034] Where w and b are the convolutional layer parameters, h is the convolutional kernel width, and x is the convolutional kernel width. i Let f be the representation of the i-th character's pinyin in the text after passing through a convolutional layer, e represents the initial representation of the character, and f is the activation function.

[0035] Steps 1-3: Obtaining glyph information.

[0036] The main structural components of Chinese characters are used as radicals. First, the main structural components of Chinese characters are crawled, and then they are combined in sequence. Finally, they are encoded into Radical Embedding using CNN.

[0037] Specifically, the main structural elements of Chinese characters in the Xinhua Dictionary can be used as the radicals or components of Chinese characters.

[0038] Step 2: Cross-Transformer-based interactive embedding, including embedding vector initialization and embedding interactive embedding.

[0039] Step 2-1: Initialize the Embedding vector.

[0040] In step 1, the Chinese characters Lattice, Pinyin, and Radical Embedding were obtained.

[0041] The input to the Cross-Transformer is obtained through a linear Transformer module:

[0042]

[0043] Among them, E L1 / L2 E P E R These are the Chinese character embeddings (Lattice, Pinyin, and Radical) obtained in step 1; I is the identity matrix, and each linear mapping matrix W represents a learnable parameter. L1 and L2 represent Lattice1 and Lattice2, which interact with Pinyin and Radical, respectively. P represents Pinyin, and R represents Radical. T represents the matrix transpose. Q, K, and V represent the three input embeddings of the Cross-Transformer.

[0044] Step 2-2: Embedding interaction embedding to obtain the input of the Cross-Transformer. Then, the Cross-Transformer network is used to perform interaction embedding calculations between Lattice and Pinyin, and between Lattice and Radical, as follows:

[0045] Att P (Q L1 ,K P V P ) = Softmax(Q L1 ,K P V P (3)

[0046] Att L1 (Q P ,K L1 V L1 ) = Softmax(Q P ,K L1 V L1 (4)

[0047] Att L2 (Q R ,K L2 V L2 ) = Softmax(Q R ,K L2 V L2 (5)

[0048] Att R (Q L2 ,K R V R ) = Softmax(Q L2 ,K R V R (6)

[0049] Where Att is the final Attention Embedding. L1 and L2 represent Lattice1 and Lattice2, which interact with Pinyin and Radical, respectively, where P represents Pinyin and R represents Radical. Q, K, and V represent the three input embeddings of the Cross-Transformer.

[0050] Step 3: Selective fusion. This includes fusion information at both the character-level and sentence-level granularities.

[0051] Step 3-1: Character granularity weight selection.

[0052] Step 2 yielded the Attention vectors for each modality. Here, this invention uses h... i Let g represent the attention vector of the i-th character. A selective gate unit is used to control how much information flows to the mixed embedding representation; the gate value is calculated by a fully connected layer and a sigmoid function. The input includes the Cross Attention representation after passing through the Cross-Transformer. The gate values ​​for Pinyin, the two Lattice, and the Radical pattern are represented as g. p g l1 g l2 g r Then the fusion embedding of the i-th character is calculated as follows:

[0053]

[0054]

[0055]

[0056]

[0057] Among them, W p W l1 W l2 W r ,b p ,b l1 ,b l2 ,b r All of these are learnable parameters. σ is the sigmoid function. Let represent the attention vectors of the i-th character in Pinyin, Lattice1, Lattice2, and Radical, respectively.

[0058] Step 3-2: Merge Embedding representations.

[0059] After obtaining the gate values ​​for each embedding in step 3-1, the attention embeddings are weighted and summed with the gate values ​​to obtain the fused embedding representation.

[0060]

[0061] Step 3-3: Sentence-level learning.

[0062] A Transformer Layer is applied to fully learn Lattice, Pinyin, and Radical information at the sentence level. The mixed representations of all characters are packaged into:

[0063]

[0064] Where H0 represents a mixed representation of all characters, This represents the fusion representation of the Nth character. This represents the fusion representation of the Mth character.

[0065] The final hybrid embedding representation is calculated as follows:

[0066] H = Transformer(H0) (13)

[0067] Here, H represents the hidden layer output after passing through the Transformer layer.

[0068] Step 4: Output the final label prediction probability.

[0069] After the fusion process in step 3 is completed, the word part is masked, and the fused information is passed to the Conditional Random Field (CRF) to obtain the final label prediction probability.

[0070] Example

[0071] like Figure 1 As shown, this invention comprises four modules. The first layer is the input layer, which converts the input text into Lattice, Pinyin, and Radical embeddings using pre-trained word vectors and an encoder. Then, the second layer, a Cross-Transformer network, interactively calculates Attention between the Lattice, Pinyin, and Radical embeddings. The resulting Attention scores are then processed by a selective fusion module to obtain a mixed vector. Finally, an output layer and a CRF (Conditional Random Field) are used to obtain the probability of the final label.

[0072] Specifically, the steps include the following:

[0073] Step 1: Convert the input text into Lattice, Pinyin, and Radical embedding representations;

[0074] Step 1 includes the following sub-steps:

[0075] Step 1-1: Lattice embedding generation.

[0076] like Figure 2 As shown, word lattice information is first obtained through dictionary matching. For example, for "Nanjing Yangtze River Bridge," word lattice information such as "Nanjing," "Nanjing City," "Mayor," "Yangtze River," "Yangtze River Bridge," and "Bridge" can be obtained. Then, relative position encoding derived from the beginning and end positions and pre-trained word vectors are used to obtain the lattice embedding.

[0077] Step 1-2: Pinyin embedding generation.

[0078] like Figure 3 As shown, the initial consonant, final vowel, and tone sequence of each Chinese character are obtained using the pypinyin library, and then the information is extracted through CNN encoding to obtain the Pinyin embedding representation of the Chinese character.

[0079] Steps 1-3: Radical embedding generation; similar to steps 1-2, such as... Figure 3 As shown, the main body of the Chinese character structure obtained in advance is used as the character shape information to obtain the Radical sequence, and then the Radical embedding representation is obtained through CNN encoding.

[0080] Step 2: Calculate interaction attention based on Cross-Transformer;

[0081] The Cross-Transformer-based interaction attention calculation in this invention is as follows: Figure 4 As shown, through four Transformer encoder layers, cross-Attention is calculated pairwise between Lattice and Pinyin, and between Lattice and Radical. The main purpose is to obtain the interaction Attention representation between Chinese characters and Pinyin, and between Chinese characters and Radical.

[0082] Step 2 includes the following sub-steps:

[0083] Step 2-1: Embedding initialization; For the embedding obtained from the input layer, a linear Transformer module is used to initialize the input of the Cross-Transformer.

[0084] Step 2-2: Attention Calculation; After obtaining the input to the Cross-Transformer, according to... Figure 4As shown, four Transformer Layers are designed. The Q values ​​of Lattice and Pinyin, and Lattice and Radical are swapped, and Attention is calculated to obtain four Attention embeddings.

[0085] Step 3: Selective fusion module.

[0086] The selective fusion module in this invention, such as Figure 5 As shown:

[0087] The selective fusion module mainly performs a dynamic fusion representation on each Attention embedding obtained by the Cross-Transformer network in step 2. First, it uses four gates to control how much information in each Attention embedding can flow into the fusion vector. After obtaining the fusion vector representation, it passes through a Transformer Layer to obtain sentence-level Attention calculations, and finally obtains a fusion embedding.

[0088] Step 3 includes the following sub-steps:

[0089] Step 3-1: Calculate the threshold values ​​for each modality.

[0090] A selective gate unit is used to control how much information can flow to the fusion embedding representation. If the current text is more conversational, more information from Pinyin Attention will flow into the fusion embedding. Conversely, if the current text is more formal, more information from Radical Attention will flow into the fusion embedding.

[0091] Step 3-2: Character-level fusion embedding.

[0092] After obtaining the gate values ​​for each modality information in step 3-1, the gate values ​​are used as weights to perform a weighted summation of the Attention function as a character-level fusion embedding.

[0093] Step 3-3: Sentence-level learning.

[0094] To enable the invention to fully learn Lattice, Pinyin, and Radical information at the sentence level, a Transformer Layer was applied later for learning.

[0095] Step 4: Model output layer.

[0096] After step 3, the fused embedding is obtained. Then, the word part is masked and passed to the Conditional Random Field (CRF) module to output the final label prediction probability.

[0097] The following tables show the experimental results of the proposed method on four public datasets. VisPhone represents the method proposed in this invention, while the others represent some classic or state-of-the-art NER models. It can be seen that the proposed method shows significant improvements over the current best models on all four datasets.

[0098] Table 1 shows the experimental results of the method proposed in this invention on the Weibo dataset.

[0099]

[0100] Table 2 shows the experimental results of the method proposed in this invention on the resume dataset.

[0101]

[0102] Table 3 shows the experimental results of the proposed method on the Ontonotes dataset.

[0103]

[0104]

[0105] Table 4 shows the experimental results of the method proposed in this invention on the MSRA dataset.

[0106]

[0107] To illustrate the content and implementation method of this invention, a specific embodiment is provided in this specification. The details introduced in the embodiment are not intended to limit the scope of the claims, but rather to aid in understanding the method described in this invention. Those skilled in the art will understand that various modifications, variations, or substitutions to the steps of the preferred embodiment are possible without departing from the spirit and scope of this invention and its appended claims. Therefore, this invention should not be limited to the content disclosed in the preferred embodiment and the accompanying drawings.

Claims

1. A method for Chinese named entity recognition based on selective fusion of multimodal information, characterized in that, First, the semantic information of words in the original text is obtained using a dictionary matching method, and the relative position encoding derived from the beginning and end positions is used to adapt to the word boundary information. Then, for each character in the text, a CNN network is used to extract the pinyin and radical information of the Chinese character; after obtaining the pinyin, radical and semantic lattice information of the Chinese character, a Cross Transformer is used to enhance the semantic information of lattice and pinyin, and lattice and radical. Then, these vectors are integrated using a selective fusion method, including fusion information at both the character granularity and sentence granularity levels; The selective fusion method is as follows: Step 1: Character granularity weight selection; Use h i Let g represent the attention vector of the i-th character; a selective gate unit is used to control how much information can flow to the mixed embedding representation, and the gate value is calculated by a fully connected layer and a sigmoid function; the input includes the Cross Attention representation after passing through the Cross-Transformer; the gate values ​​for Pinyin, the two Lattice, and Radical modes are represented as g. p g l1 g l2 g r The fusion embedding of the i-th character is calculated as follows: Among them, W p W l1 W l2 W r ,b p ,b l1 ,b l2 ,b r All of these are learnable parameters; σ is the sigmoid function; Let represent the attention vectors of the i-th character in Pinyin, Lattice1, Lattice2, and Radical, respectively. Step II: Merge Embedding representations; After obtaining the gate values ​​for each embedding in step I, the attention embeddings are weighted and summed with the gate values ​​to obtain the fused embedding representation. Step III: Sentence-level granularity learning; Applying a Transformer Layer, we fully learn Lattice, Pinyin, and Radical information at the sentence level; the mixed representations of all characters are packaged into: Where H0 represents a mixed representation of all characters, This represents the fusion representation of the Nth character. This represents the fusion representation of the Mth character; The final hybrid embedding representation is calculated as follows: H = Transformer(H0) (13) Where H represents the hidden layer output after passing through the Transformer layer; Finally, the word portion is masked, and the fused information is passed to a conditional random field to obtain the final label prediction probability, thus completing Chinese named entity recognition.

2. The Chinese named entity recognition method based on selective fusion of multimodal information as described in claim 1, characterized in that: The method for obtaining semantic Lattice information is as follows: First, the semantic information of Chinese characters is obtained using dictionary matching, and the relative position encoding derived from the head and tail positions is used to adapt to the word boundary information; then, Lattice Embedding is obtained by initializing with pre-trained word vectors. Obtain the pinyin information, including the initials, finals, and tones of the Chinese characters. Combine the pinyin of each Chinese character in the order of initials, finals, and tones, and then input it into a CNN convolutional network for embedding representation according to Equation 1. x i =f(w·e i:i+h-1 +b) (1) Where w and b are the convolutional layer parameters, h is the convolutional kernel width, and x is the convolutional kernel width. i Let f be the representation of the i-th character's pinyin in the text after passing through a convolutional layer, e be the initial representation of the character, and f be the activation function; To obtain character shape information, the main body of the Chinese character structure is used as the radical of the Chinese character. First, the main body of the Chinese character structure is crawled, the main body of the Chinese character structure is combined in sequence, and then CNN is used to encode it into Radical Embedding.

Images