FLAT-based Chinese named entity recognition method, storage medium and equipment
By introducing a local window attention mechanism in the Flat model, it is improved to global-local window self-attention mechanism, solving the problem of calculating efficiency when processing long sequences when processing long sequences, and realizing more efficient calculations and better context information information Capture ability.
Patent Information
- Application Number
- CN202411394963.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-08
- Publication Date
- 2025-01-28
AI Technical Summary
The global self-attention mechanism of the FLAT model is inefficient in processing long sequences and may ignore local context information.
Introduce a local window attention mechanism to improve the Transformer encoder as a global-local window self-attention mechanism. Through the scope of attention to restrictions on local windows, reduce the complexity of calculation, and combine global attention to capture long-distance dependencies.
Improving the efficiency of calculation, enhancing the capture ability of local context information, ensuring that the model can efficiently handle the complex dependency relationship in the long sequence text, and improve the generalization and robustness of the model.
Smart Images
Figure CN119378549A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of Chinese named entity recognition, and in particular to a Chinese named entity recognition method, storage medium, and device based on FLAT. Background Art
[0002] Chinese named entity recognition (NER) is a key task in natural language processing, which aims to identify named entities in text, such as people, geographic locations, organization names, time and currency. NER plays an important role in multiple downstream tasks of natural language processing, such as recommendation systems, machine translation, knowledge graphs and semantic search. Traditional Chinese NER tasks usually face the challenge of word segmentation error propagation.
[0003] Early named entity recognition methods mainly include rule-based matching methods, statistical machine learning methods, and neural network deep learning methods. Rule-based matching methods rely on hand-crafted semantic and syntactic rules to identify entities and do not require annotated data. When the dictionary is exhaustive, the rule-based matching model performs well, but the generalization ability of this method is poor. When it is switched to other specific fields, the recognition results will be greatly affected. Different rules need to be built for different fields, resulting in low efficiency. The recognition effect of statistical machine learning methods depends on a large amount of annotated corpus, which limits the expansion and migration of its application scope. With the increase of annotated data, the training cost of the model will also increase significantly. In the neural network deep learning-based methods, characters and words are equally important for Chinese NER, but modeling based on only one of them cannot fully explore the contextual information hidden in the sentence sequence. There is no space between Chinese characters for segmentation, and errors in word segmentation will affect the recognition effect. In addition, Chinese has the problem of polysemy. The pre-training models such as Word2Vec used by many scholars mainly focus on the features between words or characters, without considering the context of words, and still cannot solve the problem of polysemy.
[0004] The FLAT (Flat-Lattice Transformer) model effectively utilizes the lattice information and demonstrates excellent parallel computing capabilities by converting the character-word lattice structure into a flat structure and introducing special position encoding. However, the global self-attention mechanism may have low computational efficiency when processing long sequences and may ignore the importance of local context information. Although the FLAT model solves the problems of word segmentation error propagation and long-distance dependency modeling to a certain extent, its global self-attention mechanism is computationally inefficient when processing long sequences and may ignore the importance of local context information. Summary of the invention
[0005] The purpose of the present invention is to solve the problem that the global self-attention mechanism of the FLAT model has low computational efficiency and ignores local context information when processing long sequences, and proposes a Chinese named entity recognition method based on FLAT, which includes the following steps:
[0006] S1. Obtain Chinese text to be recognized;
[0007] S2. Based on the FLAT model, on the basis of the global attention mechanism in the Transformer encoder of the FLAT model, a local window attention mechanism is introduced to construct an improved FLAT model of the Transformer encoder with an improved global-local window self-attention mechanism;
[0008] S3. Input the Chinese text to be recognized into the improved FLAT model. After passing through the embedding layer of the improved FLAT model, the input Chinese text is converted into an embedded representation. The embedded representation is encoded using the improved Transformer encoder to generate a context-related representation. The context-related representation is decoded, and label prediction is performed through the conditional random field layer of the improved FLAT model. Finally, the named entity recognition result is output.
[0009] Furthermore, the attention score function of the global-local window self-attention mechanism is expressed as:
[0010]
[0011] in, represents the attention score of the global-local window self-attention mechanism, α represents the weight parameter, represents the global attention score, represents the score of local attention, W q Represents the learnable weight matrix of the query vector, W k,E ,W k,R They represent the learnable weight matrices of the key vector with respect to the word embedding matrix and the relative position encoding, E represents the word embedding matrix, R represents the relative position encoding, u and v represent the learnable bias vectors, respectively. denote the input vectors at positions i and j, respectively, R ij represents the relative position encoding reflecting the relative distance between positions i and j, and w represents the window range.
[0012] Furthermore, in S3, the Chinese text to be recognized is input into the improved FLAT model, and the input Chinese text is converted into an embedded representation through the embedding layer of the improved FLAT model, specifically:
[0013] Convert the embedded representation of the input Chinese text into a character sequence and generate a lattice structure based on the vocabulary;
[0014] Convert the word structure into a plane structure and determine the head and tail positions of each character and word in the word structure;
[0015] Assign head and tail position codes to each character and word, and use relative position coding method to represent the relative relationship between characters and words.
[0016] Furthermore, the relative position encoding is expressed as:
[0017]
[0018] in, Indicates the distance between the head position of the i-th word and the head position of the j-th word, head i Indicates the head position of the i-th word, head j represents the head position of the jth word, Indicates the distance between the tail position of the i-th word and the tail position of the j-th word, tail i Indicates the tail position of the i-th word, tail j represents the tail position of the jth word, R ij represents the relative position encoding reflecting the relative distance between positions i and j, W r represents the learnable parameters, express The embedding vector of Represents a splicing operation, express The embedding vector of express The embedding vector of
[0019] when or When is an even number, the embedding vector is expressed as:
[0020]
[0021] when or When is an odd number, the embedding vector is expressed as:
[0022]
[0023] Indicates when or The position encoding is the embedding vector when the position is an even position, Indicates when or The position encoding is the embedding vector when the odd position, d ij express or k represents the dimension index of the positional encoding, and takes values of 0, 1, 2, ..., d model =H×d head , d head Represents the dimensions of each head.
[0024] Furthermore, the improved Transformer encoder includes a global-local window self-attention mechanism and a feed-forward neural network layer; the feed-forward neural network layer is expressed as:
[0025] FFN(x)=ReLU(W1x+b1)W2+b2
[0026] Among them, FFN(x) represents the feedforward neural network layer with input x, W1 and W2 represent weight parameters, b1 and b2 represent bias parameters, and ReLU represents the ReLU activation function.
[0027] Furthermore, label prediction is performed through the conditional random field layer of the improved FLAT model, which is expressed as:
[0028]
[0029] Among them, X represents the input sequence, Y represents the label sequence, Indicates that from label y i-1 Move to label y i The score, Indicates the yth row in the i-th row i The prediction score for the column position.
[0030] The present invention also proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned FLAT-based Chinese named entity recognition method is implemented.
[0031] The present invention also proposes an electronic device, comprising a processor and a memory, wherein the processor and the memory are interconnected, wherein the memory is used to store a computer program, the computer program comprises computer-readable instructions, and the processor is configured to call the computer-readable instructions to execute the above-mentioned FLAT-based Chinese named entity recognition method.
[0032] The beneficial effects brought by the technical solution provided by the present invention are:
[0033] The present invention proposes a Chinese named entity recognition method based on FLAT. On the basis of the original FLAT model, a local window attention mechanism is introduced, and the self-attention mechanism in the Transformer encoder is improved to a global-local window self-attention mechanism. The local window attention limits the scope of attention and only calculates the attention weight within the local window, thereby greatly reducing the computational complexity and resource consumption, improving the computational efficiency, and ensuring that the model effectively captures local context information. The global attention mechanism is used to capture long-distance dependencies to ensure that the model can understand the overall context information. The combination of the two enables the model to process efficiently and comprehensively cope with complex dependencies in long sequence texts. This mechanism is not only applicable to Chinese named entity recognition tasks, but can also be extended to other natural language processing tasks such as text classification and machine translation. By enhancing the ability to capture local context information, the model performs better in terms of generalization and robustness. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 It is the structure of the FLAT model;
[0035] Figure 2 is a flow chart of a Chinese named entity recognition method based on FLAT according to an embodiment of the present invention;
[0036] Figure 3 is a structural diagram of an improved FLAT model according to an embodiment of the present invention;
[0037] Figure 4 It is a block diagram of an electronic device in an exemplary embodiment of the present invention. DETAILED DESCRIPTION
[0038] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described below with reference to the accompanying drawings.
[0039] The present invention takes the FLAT model (from: FLAT: Chinese NERUsing Flat-Lattice Transformer) as the theoretical basis. The structure of the FLAT model is as follows: Figure 1 As shown in the figure, it includes an embedding layer (Embedding), a Transformer encoder, the Transformer encoder consists of a self-attention layer and a feedforward neural network layer, the self-attention layer consists of a self-attention mechanism (self-Attention) and residual connection & normalization (Add&Norm), and the feedforward neural network layer consists of a feedforward neural network (Feedforward neural network: FFN) and residual connection & normalization (Add&Norm).
[0040] The flowchart of the Chinese named entity recognition method based on FLAT in the embodiment of the present invention is referenced Figure 2 , specifically including the following steps:
[0041] S1. Obtain Chinese text to be recognized.
[0042] First, obtain the Chinese text data to be processed and convert it into a character sequence. The text input T = {c_1, c_2, ..., c_n}, where c_i represents the i-th character. Then, by searching the vocabulary, match each character to a potential word and build a word lattice structure L = {w_1, w_2, ..., w_m}, where w_j represents the j-th potential word. This process prepares for the subsequent embedding and encoding steps.
[0043] S2. Based on the FLAT model, on the basis of the global attention mechanism in the Transformer encoder of the FLAT model, a local window attention mechanism is introduced to construct an improved FLAT model of the Transformer encoder with an improved global-local window self-attention mechanism. Figure 3 , Figure 3 It is a structural diagram of an improved FLAT model according to an embodiment of the present invention.
[0044] Specifically, the attention score function of the global-local window self-attention mechanism is expressed as:
[0045]
[0046] in, represents the attention score of the global-local window self-attention mechanism, α represents the weight parameter, represents the global attention score, represents the score of local attention, W q Represents the learnable weight matrix of the query vector, W k,E ,W k,R They represent the learnable weight matrices of the key vector with respect to the word embedding matrix and the relative position encoding, E represents the word embedding matrix, R represents the relative position encoding, u and v represent the learnable bias vectors, respectively. denote the input vectors at positions i and j, respectively, R ij represents the relative position encoding reflecting the relative distance between positions i and j, and w represents the window range.
[0047] By setting the local window size w, the model calculates the attention scores between adjacent characters or words in addition to the global attention. The weights of global and local attention are regulated by the parameter α to ensure that both short-range and long-range dependencies are captured, thereby improving the understanding and processing capabilities of complex text contexts.
[0048] S3. Input the Chinese text to be recognized into the improved FLAT model. After passing through the embedding layer of the improved FLAT model, the input Chinese text is converted into an embedded representation. The embedded representation is encoded using the improved Transformer encoder to generate a context-related representation. The context-related representation is decoded, and label prediction is performed through the conditional random field layer of the improved FLAT model. Finally, the named entity recognition result is output.
[0049] Specifically, the model first converts characters and words into vector representations through an embedding layer, then converts the embedded representation of the input Chinese text into a character sequence, and generates a lattice structure in combination with the vocabulary; the lattice structure is then converted into a planar structure, and the head and tail positions of each character and word in the lattice structure are determined; then each character and word is assigned a head and tail position code, and the relative position coding method is used to represent the relative relationship between characters and words.
[0050] The relative position encoding is expressed as:
[0051]
[0052] in, Indicates the distance between the head position of the i-th word and the head position of the j-th word, head i Indicates the head position of the i-th word, head j represents the head position of the jth word, Indicates the distance between the tail position of the i-th word and the tail position of the j-th word, tail i Indicates the tail position of the i-th word, tail j represents the tail position of the jth word, R ij represents the relative position encoding reflecting the relative distance between positions i and j, W r represents the learnable parameters, express The embedding vector of Represents a splicing operation, express The embedding vector of express The embedding vector of .
[0053] when or When is an even number, the embedding vector is expressed as:
[0054]
[0055] when or When is an odd number, the embedding vector is expressed as:
[0056]
[0057] Indicates when or The position encoding is the embedding vector when the position is an even position, Indicates when or The position encoding is the embedding vector when the odd position, d ij express or k represents the dimension index of the positional encoding, and takes values of 0, 1, 2, ..., d model =H×d head , d head Represents the dimensions of each head.
[0058] Next, the improved Transformer encoder combines global attention and local window self-attention mechanisms to encode the embedding vector and generate context-dependent representations.
[0059] The feedforward neural network layer is represented as:
[0060] FFN(x)=ReLU(W1x+b1)W2+b2
[0061] Among them, FFN(x) represents the feedforward neural network layer with input x, W1 and W2 represent weight parameters, b1 and b2 represent bias parameters, and ReLU represents the ReLU activation function.
[0062] The encoded representation is passed to the conditional random field layer, and the label prediction for named entity recognition is performed by calculating the maximum score of the sequence label. Label prediction is performed through the conditional random field layer of the improved FLAT model, which is expressed as:
[0063]
[0064] Among them, X represents the input sequence, Y represents the label sequence, Indicates that from label y i-1 Move to label y i The score, Indicates the yth row in the i-th row i The prediction score for the column position.
[0065] Finally, the model outputs named entity recognition results containing entity boundaries and types.
[0066] In an exemplary embodiment, a computer-readable storage medium is included, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned FLAT-based Chinese named entity recognition method is implemented.
[0067] See also Figure 4 In an exemplary embodiment, an electronic device is also included, including at least one processor, at least one memory, and at least one communication bus.
[0068] A computer program is stored in the memory, and the computer program includes computer-readable instructions. The processor calls the computer-readable instructions stored in the memory through a communication bus to execute the above-mentioned FLAT-based Chinese named entity recognition method.
[0069] The embodiment of the present invention takes the performance of Chinese named entity recognition model as the research object, and conducts experiments on the most commonly used Weibo dataset (from: Named entity recognition for Chinese social media with jointly trained embeddings), Resume dataset (from: Chinese NER using lattice LSTM) and OntoNotes 4.0 dataset (from: OntoNotes Release 5.0) for evaluating Chinese named entity recognition model. Table 1 shows the statistical data of these datasets.
[0070] Table 1
[0071] Dataset Ontonotes Weibo Resume Training set size 15740 1350 3821 Average number of characters 36.92 54.37 32.15 Average number of words 17.59 21.49 24.99 Average number of entities 1.15 1.42 3.48
[0072] At the same time, several existing Chinese named entity recognition models were selected for comparison:
[0073] (1) TENER model (from: TENER: Adapting Transformer Encoder for Named Entity Recognition), this method uses Transformer Encoder for named entity recognition. By introducing relative position encoding, it can better capture the dependencies between words, thereby improving the performance of the model when processing long sequences of text. TENER achieves accurate recognition of complex text by applying a multi-head self-attention mechanism on character-level and word-level embeddings.
[0074] (2) LatticeLSTM model (from: Chinese NER Using Lattice LSTM), by combining the lattice structure of characters and words, uses the LSTM network to capture the relationship between characters and words. This method can effectively deal with word segmentation ambiguity and word boundary problems in Chinese text, thereby improving the accuracy of named entity recognition. LatticeLSTM introduces vocabulary information into the input sequence, enhancing the ability to capture word boundaries and context.
[0075] (3) FLAT model: By converting the character-word lattice structure into a planar structure and introducing special position encoding, the FLAT model effectively utilizes the lattice information and demonstrates excellent parallel computing capabilities.
[0076] The experiment used the F1 index as the evaluation indicator, and the final test results are shown in Table 2.
[0077] Table 2
[0078]
[0079]
[0080] It can be seen from Table 2 that the model proposed in the present invention is superior to the compared benchmark model in terms of evaluation indicators.
[0081] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A Chinese named entity recognition method based on FLAT, characterized in that: The following steps are involved: S1. Obtain Chinese text to be recognized; S2. Based on the FLAT model, the self-attention mechanism in the Transformer encoder of the FLAT model is replaced with a global-local window self-attention mechanism to construct an improved FLAT model of the improved Transformer encoder; S3. Input the Chinese text to be recognized into the improved FLAT model. After passing through the embedding layer of the improved FLAT model, the input Chinese text is converted into an embedded representation. The embedded representation is encoded using the improved Transformer encoder to generate a context-related representation. The context-related representation is decoded, and label prediction is performed through the conditional random field layer of the improved FLAT model. Finally, the named entity recognition result is output.
2. The Chinese named entity recognition method based on FLAT according to claim 1 is characterized in that: The attention score function of the global-local window self-attention mechanism is expressed as: in, represents the attention score of the global-local window self-attention mechanism, α represents the weight parameter, represents the global attention score, represents the score of local attention, W q Represents the learnable weight matrix of the query vector, W k,E ,W k,R They represent the learnable weight matrices of the key vector with respect to the word embedding matrix and the relative position encoding, E represents the word embedding matrix, R represents the relative position encoding, u and v represent the learnable bias vectors, respectively. denote the input vectors at positions i and j, respectively, R ij represents the relative position encoding reflecting the relative distance between positions i and j, and w represents the window range.
3. The Chinese named entity recognition method based on FLAT according to claim 1 is characterized in that: In S3, the Chinese text to be recognized is input into the improved FLAT model. After the embedding layer of the improved FLAT model, the input Chinese text is converted into an embedded representation, specifically: Convert the embedded representation of the input Chinese text into a character sequence and generate a lattice structure based on the vocabulary; Convert the word structure into a plane structure and determine the head and tail positions of each character and word in the word structure; Assign head and tail position codes to each character and word, and use relative position coding method to represent the relative relationship between characters and words.
4. The Chinese named entity recognition method based on FLAT according to claim 3 is characterized in that: The relative position encoding is expressed as: in, Indicates the distance between the head position of the i-th word and the head position of the j-th word, head i Indicates the head position of the i-th word, head j represents the head position of the jth word, Indicates the distance between the tail position of the i-th word and the tail position of the j-th word, tail i Indicates the tail position of the i-th word, tail j represents the tail position of the jth word, R ij represents the relative position encoding reflecting the relative distance between positions i and j, W r represents the learnable parameters, express The embedding vector of Represents a splicing operation, express The embedding vector of express The embedding vector of when or When is an even number, the embedding vector is expressed as: when or When is an odd number, the embedding vector is expressed as: Indicates when or The position encoding is the embedding vector when the position is an even position, Indicates when or The position encoding is the embedding vector when the odd position, d ij express or k represents the dimension index of the positional encoding, and takes values of 0, 1, 2, ..., d model =H×d head , d head Represents the dimensions of each head.
5. The Chinese named entity recognition method based on FLAT according to claim 2 is characterized in that: The improved Transformer encoder includes a global-local window self-attention mechanism and a feed-forward neural network layer; the feed-forward neural network layer is expressed as: FFN(x)=ReLU(W1x+b1)W2+b2 Among them, FFN(x) represents the feedforward neural network layer with input x, W1 and W2 represent weight parameters, b1 and b2 represent bias parameters, and ReLU represents the ReLU activation function.
6. The Chinese named entity recognition method based on FLAT according to claim 1 is characterized in that: Label prediction is performed through the conditional random field layer of the improved FLAT model, expressed as: Among them, X represents the input sequence, Y represents the label sequence, Indicates that from label y i-1 Move to label y i The score, Indicates the yth row in the i-th row i The prediction score for the column position.
7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
8. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the processor and the memory are interconnected, wherein the memory is used to store a computer program, the computer program comprises computer-readable instructions, and the processor is configured to call the computer-readable instructions to execute the method according to any one of claims 1 to 6.