Chinese named entity recognition method, electronic device, and storage medium

Abstract

The application provides a Chinese named entity recognition method based on an attention mechanism, an electronic device and a storage medium, which comprises the following steps: inputting a text to be recognized into an embedding layer to obtain a word vector; using a Transformer encoder to extract features of the word vector to obtain a first context feature; using a Bi-LSTM model to extract features of the word vector to obtain a second context feature; fusing the first context feature and the second context feature to obtain a fused feature; and decoding the fused feature to obtain Chinese named entities corresponding to the text to be recognized. The Chinese named entity recognition method based on the attention mechanism realizes deep fusion of global semantic information and directional information. In order to obtain more context information and solve the problem of polysemy, a RoBERTa-wwm pre-training model is used as a character-level embedding, so that the model recognition effect is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a Chinese named entity recognition method, electronic device, and storage medium based on an attention mechanism. Background Technology

[0002] Named Entity Recognition (NER) is a task that aims to determine the boundaries of entities in text and accurately classify them. NER is fundamental to many Natural Language Processing (NLP) tasks, such as information extraction, question answering, information retrieval, and knowledge graphs, and has therefore received widespread attention and research in the field of NLP.

[0003] Named Entities (NEs) were first used at the 6th Conference on Information Understanding (MUC-6) to identify names of people, places, and organizations in text. Early methods for NE recognition were rule-based and statistical learning-based. While traditional methods achieved some success, they suffered from difficulties in constructing manual features, heavy reliance on domain-specific knowledge, and challenges in transferring model transferability. In recent years, with the continuous growth of computing power, deep learning-based NE recognition methods have gradually replaced traditional methods and become the mainstream approach.

[0004] In deep learning-based named entity recognition methods, named entity recognition is generally regarded as a sequence labeling task. Its general framework consists of three parts: distributed representations of input, a context encoder, and a tag decoder.

[0005] With the development and application of natural language processing technology in the Chinese language field, Chinese named entity recognition has also attracted widespread attention from scholars both domestically and internationally. Compared with English, there are no clear boundaries between Chinese words, making Chinese named entity recognition more challenging. Initially, Chinese named entity recognition tasks required Chinese word segmentation (CWS) first, followed by word-level vector representations as input. Subsequent research showed that character-level embeddings typically outperform word-level vector representations. This is because Chinese word segmentation inevitably introduces errors, leading to problems in predicting entity boundaries. Therefore, using character-level embeddings as input for Chinese named entity recognition methods has become the preferred choice for researchers.

[0006] Although the character-level embedding method avoids the errors caused by Chinese word segmentation, it also loses lexical information. Since entity boundaries are generally the boundaries of words, this poses challenges to the prediction of entity boundaries, and the most popular BERT pre-trained model in recent years cannot solve this problem either. Because the BERT model segments text at the character level rather than at the word level, only character-level embeddings can be obtained using the BERT pre-trained model, and Chinese lexical information cannot be obtained. As an improved model of BERT, the RoBERTa-wwm model uses a whole-word masking strategy and a dynamic masking strategy on Chinese texts, so that the model obtains certain lexical information and more semantic information, making it more suitable for Chinese named entity recognition tasks.

[0007] While breakthroughs have been made in the research on vector representation of Chinese texts, the context encoder, as an important part of Chinese named entity recognition, has also advanced rapidly. The Bidirectional Long-Short-Term Memory (BiLSTM), the most commonly used context encoder in Chinese named entity recognition tasks, has good sequence modeling capabilities, can make good use of context features, and retain the order information of the text. Although BiLSTM uses a forward and backward network architecture, it only concatenates the forward and backward information. For a certain character or word in the sequence, it is not simultaneously associated with the context, lacking global information. In recent years, due to its good parallelism and global modeling capabilities, the Transformer network has been very popular in natural language processing tasks such as machine translation and pre-trained models, and the Transformer architecture and attention mechanism have also been applied to named entity recognition tasks. However, the Transformer architecture pays more attention to global semantics and is insensitive to position and direction information. In named entity recognition tasks, the relative direction and distance between characters are very important and help the model identify entities. For example, the words after 'in' are often entities of the location category. Therefore, it is necessary to improve the Transformer encoder to make it suitable for Chinese named entity recognition tasks.

[0008] The above traditional named entity recognition methods have the following technical problems:

[0009] For Transformer, it is not easy to distinguish whether the context information comes from the above or below text.

[0010] BiLSTM can collect context information from the above and below text of the current word or character in a distinguishable way, but lacks global semantic information. Summary of the Invention

[0011] This application aims to provide a Chinese named entity recognition method, electronic device, and storage medium based on an attention mechanism, in order to solve the problem of low accuracy in existing Chinese named entity recognition technologies.

[0012] On the one hand, this application provides a Chinese named entity recognition method based on an attention mechanism, wherein the text to be recognized is input into an embedding layer to obtain word vectors;

[0013] The word vectors are extracted using a Transformer encoder to obtain the first context features;

[0014] The word vectors are used to extract features to obtain the second context features;

[0015] The first context feature and the second context feature are fused to obtain the fused feature;

[0016] The fused features are decoded to obtain the Chinese named entities corresponding to the text to be identified.

[0017] Furthermore, the step of using a Transformer encoder to extract features from the word vectors to obtain the first contextual features includes:

[0018] The relative position encoding of each character is obtained using a preset calculation formula;

[0019] The word vectors and relative position codes are analyzed using a Transformer encoder to obtain attention scores, thereby obtaining the first contextual features.

[0020] Furthermore, the second contextual features obtained by using the Bi-LSTM model to extract features from the word vectors include:

[0021] The Bi-LSTM model is used to input the word vectors with forward and backward directions respectively. The forward and backward vectors are calculated and then concatenated to obtain the second contextual feature.

[0022] Furthermore, the fusion of the first context feature and the second context feature to obtain the fused feature includes:

[0023] An attention score is obtained using an attention mechanism, and the attention score is used as a fusion parameter.

[0024] Dynamically allocate the first fusion parameter of the first context feature and the second fusion parameter of the second context feature;

[0025] The first context feature and the second context feature are dynamically fused according to the first fusion parameter and the second fusion parameter.

[0026] Furthermore, decoding the fused features includes:

[0027] The fused features are decoded using a conditional random field, and corresponding labels are output.

[0028] Furthermore, the step of inputting the text to be recognized into the embedding layer to obtain word vectors includes:

[0029] The text to be identified is analyzed to obtain character-level embedding and binary character-level word embedding. Character-level embedding is based on characters, while binary character-level word embedding is based on pairs of characters.

[0030] Word vectors are obtained by concatenating character-level embeddings and bigram character-level word embeddings.

[0031] Furthermore, Word2Vector word vectors and pre-trained models are used as embedding layers.

[0032] Furthermore, when calculating the attention score, the Transformer encoder calculates the word vectors and relative position encodings separately, and adds bias terms for relative distance and direction.

[0033] On the one hand, this application also provides an electronic device, the electronic device comprising:

[0034] One or more processors;

[0035] Memory; and

[0036] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the method described in any one of the first aspects.

[0037] In one aspect, this application also provides a computer-readable storage medium having a computer program stored thereon, the computer program being loaded by a processor to perform the steps of the method described in any of the first aspects.

[0038] Unlike existing technologies, this application provides a Chinese named entity recognition method, electronic device, and storage medium based on an attention mechanism, comprising: inputting the text to be recognized into an embedding layer to obtain word vectors; using a Transformer encoder to extract features from the word vectors to obtain first context features; using a Bi-LSTM model to extract features from the word vectors to obtain second context features; fusing the first context features and the second context features to obtain fused features; and decoding the fused features to obtain the Chinese named entity corresponding to the text to be recognized.

[0039] This application proposes an attention-based Chinese named entity recognition method. Considering that the original Transformer encoder possesses global modeling capabilities but is insensitive to positional and directional information, the method improves the Transformer encoder by using relative positional encoding and modifying the attention calculation formula. Simultaneously, BiLSTM is utilized to further enhance directional information. Finally, an attention mechanism is used to dynamically fuse contextual features, achieving deep fusion of global semantic and directional information. To obtain more contextual information and address the polysemy problem, a RoBERTa-wwm pre-trained model is used as a character-level embedding, improving the model's recognition performance. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a schematic diagram of a Chinese named entity recognition system provided in an embodiment of this application.

[0042] Figure 2 This is a schematic flowchart of an embodiment of the Chinese named entity recognition method based on the attention mechanism provided in this application.

[0043] Figure 3 This is a schematic diagram of an embodiment of the TBAC model provided in this application.

[0044] Figure 4 This is a schematic diagram of the dataset in the embodiments of this application;

[0045] Figure 5 This is a schematic diagram of hyperparameter settings in an embodiment of this application;

[0046] Figure 6This is a schematic diagram of the comparative experimental results based on Word2Vector word vectors in the embodiments of this application;

[0047] Figure 7 This is a schematic diagram of the comparative experimental results based on the pre-trained model in the embodiments of this application;

[0048] Figure 8 This is a schematic diagram of the ablation experiment results in the embodiments of this application;

[0049] Figure 9 This is a schematic diagram of an embodiment of the electronic device provided in this application. Detailed Implementation

[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] In the description of this application, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more features. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0052] In this application, the term "exemplary" is used to mean "used as an example, illustration, or description." Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use this application. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that this application can be made without using these specific details. In other instances, well-known structures and processes are not described in detail to avoid obscuring the description of this application with unnecessary detail. Therefore, this application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0053] It should be noted that since the method in this application embodiment is executed in an electronic device, the processing objects of each electronic device exist in the form of data or information, such as time, which is essentially time information. It can be understood that if size, quantity, position, etc. are mentioned in subsequent embodiments, they are all corresponding data that exist so that the electronic device can process them. Specific details will not be elaborated here.

[0054] This application provides a Chinese named entity recognition method, electronic device, and storage medium based on an attention mechanism, which will be described in detail below.

[0055] Please see Figure 1 , Figure 1 This is a schematic diagram of a Chinese named entity recognition system provided in an embodiment of this application. The system may include an electronic device 100, and the electronic device 100 may integrate a device for Chinese named entity recognition, such as... Figure 1 Electronic devices in the system.

[0056] In this embodiment, the electronic device 100 can be a standalone server, a server network, or a server cluster. For example, the electronic device 100 described in this embodiment includes, but is not limited to, a computer, a network host, a single network server, a set of multiple network servers, or a cloud server composed of multiple servers. The cloud server is composed of a large number of computers or network servers based on cloud computing.

[0057] Those skilled in the art will understand that Figure 1 The application environment shown is merely one application scenario of the solution in this application and does not constitute a limitation on the application scenario of the solution in this application. Other application environments may include more than one application scenario. Figure 1 The number of more or fewer electronic devices shown, for example Figure 1Only one electronic device is shown in the diagram. It is understood that the system may also include one or more other servers, which are not specified here.

[0058] In addition, such as Figure 1 As shown, the system may also include a memory 200 for storing data.

[0059] It should be noted that, Figure 1 The schematic diagram of the system shown is merely an example. The system and scenario described in this application are intended to more clearly illustrate the technical solutions of this application and do not constitute a limitation on the technical solutions provided in this application. As those skilled in the art will know, with the evolution of the system and the emergence of new business scenarios, the technical solutions provided in this application are also applicable to similar technical problems.

[0060] First, this application provides a Chinese named entity recognition method based on an attention mechanism. The execution subject of this method is a Chinese named entity recognition device, which is applied to an electronic device. The Chinese named entity recognition method includes: using a Transformer encoder to extract features from the word vectors to obtain first context features; using a Bi-LSTM model to extract features from the word vectors to obtain second context features; fusing the first context features and the second context features to obtain fused features; and decoding the fused features to obtain the Chinese named entities corresponding to the text to be recognized.

[0061] See Figure 2 , Figure 2 This is a schematic flowchart of one embodiment of the method provided in this application. (In conjunction with...) Figures 2-8 The method includes:

[0062] Step S201: Input the text to be recognized into the embedding layer to obtain word vectors;

[0063] In this embodiment, the text to be identified is analyzed to obtain character-level embeddings and bigram character-level word embeddings. Character-level embeddings are based on characters, while bigram character-level word embeddings are based on pairs of characters. The character-level embeddings and bigram character-level word embeddings are concatenated to obtain word vectors.

[0064] Specifically, the overall architecture of the TBAC model used in this embodiment is as follows: Figure 3As shown, the model mainly consists of four parts: an embedding layer, a context encoding layer, a feature fusion layer, and a decoding layer. The model first inputs the text into the embedding layer to obtain character-level word embeddings. Then, in the context encoding layer, it uses Transformer and Bi-LSTM to extract context features, which are then fed into the feature fusion layer and fused using an attention mechanism. Finally, in the decoding layer, a conditional random field is used to decode and output the labels.

[0065] This embodiment uses Word2Vector word vectors and a pre-trained model as the embedding layer, where the pre-trained model is a pre-trained RoBERTa-wwm model. It is assumed that the initial input to the model is a sentence. When using the RoBERTa-wwm model, the output is a vector. When using Word2Vector word vectors, the character-level embeddings and bigram character-level word embeddings are obtained as follows: and Character-level embeddings are based on individual characters, while binary character-level word embeddings are based on pairs of characters. These are then concatenated to obtain the final word vector.

[0066]

[0067] Step S202: Use a Transformer encoder to extract features from the word vectors to obtain the first context features;

[0068] In this embodiment, an improved Transformer encoder is used to extract features from the word vectors to obtain the first context features. The improved Transformer encoder calculates the attention score by separating the word vectors from the relative position encoding and adding bias terms for relative distance and direction. No scaling is performed when calculating the attention score.

[0069] The relative position encoding of each character is obtained using a preset calculation formula; the word vector and the relative position encoding are analyzed using a Transformer encoder to obtain an attention score, thereby obtaining the first context feature.

[0070] Specifically, the one-layer structure of the Transformer encoder is as follows: Figure 2 The left side of the model's context encoding layer is shown, which includes a multi-head self-attention layer, a feedforward neural network layer, and uses layer normalization and residual connections. The original Transformer encoder uses absolute encoding to generate positional codes, and the positional code of the t-th character can be represented by the following equation:

[0071]

[0072] The range of values ​​for i is: d represents the dimension of the input word vectors. The obtained positional encodings and word vectors are added position-by-position to obtain the input matrix for the multi-head self-attention layer. ,in Let H be the sequence length. Map H to... The method is as follows:

[0073]

[0074] in, All are dimension The variable weight matrix, where For hyperparameters, the scaled dot product attention can be calculated using the following equation.

[0075]

[0076] When using multi-head self-attention, the calculation formula is as follows:

[0077]

[0078]

[0079]

[0080] Where n is the total number of heads, and the superscript h represents the head index. The attention spans representing multiple heads are pieced together. It is a dimension of The variable weight matrix. The output x of the multi-head self-attention layer will be further processed by the feedforward neural network layer, which can be represented as follows:

[0081]

[0082] Here, W1, W2, b1, and b2 are learnable parameters. , , , ,in It is a hyperparameter.

[0083] In this embodiment, the original Transformer encoder is improved by employing relative position encoding and modifying the attention calculation formula. First, when mapping H to Q, K, V, ... The purpose of not performing a linear transformation is to break the symmetry and enhance distance perception. The transformation formula is as follows:

[0084]

[0085] in, The dimension is , Let be one of the feature dimensions of the head; The vector assigned to H for each head. Next, replacing the cosine function in the absolute encoding with the sine function, the new relative position encoding is as follows:

[0086]

[0087] Where t is the index of the target character, j is the index of the context character, and the value of i ranges from 1 to 1. When calculating the attention score, the word vectors and relative position encodings are calculated separately, and a bias term is added. The calculation formula is as follows:

[0088]

[0089] Among them It is the attention score for two characters; It is the deviation of the t-th character in terms of relative distance; It is the deviation of the j-th character; It is an offset term relative to distance and direction; and These are all learnable parameters.

[0090] Finally, when calculating attention, no scaling is applied, and the calculation formula is as follows:

[0091]

[0092] By employing relative position encoding and modifying the attention calculation formula as described above, the position and orientation awareness of the Transformer encoder are improved, making the Transformer suitable for Chinese named entity recognition tasks.

[0093] Step S203: Use the Bi-LSTM model to extract features from the word vectors to obtain the second context features;

[0094] In this embodiment, the Bi-LSTM model is used to input the word vectors with forward and backward directions respectively. The forward and backward vectors are calculated and then concatenated to obtain the second contextual feature.

[0095] Long Short-Term Memory (LSTM) networks are a special type of recurrent neural network (RNN). LSTM can alleviate the gradient vanishing and gradient exploding problems encountered by traditional RNNs. A forget gate is introduced into LSTM to control the information flow, thus selectively remembering information.

[0096] In Chinese named entity recognition tasks, we need not only the preceding context information but also the following context information for the target character. Therefore, we use a bidirectional long short-term memory network (BiLSTM) as the context encoder, the structure of which is as follows: Figure 3 The context encoding layer of the overall model architecture is shown on the right. BiLSTM uses both forward and backward inputs for the character-level embeddings output by the embedding layer. The forward and backward vectors are calculated, and then concatenated as the output of the hidden layer. The implementation is as follows:

[0097]

[0098]

[0099]

[0100] Step S204: Fuse the first context feature and the second context feature to obtain the fused feature;

[0101] In this embodiment, an attention score is obtained using an attention mechanism and used as a fusion parameter; a first fusion parameter for the first context feature and a second fusion parameter for the second context feature are dynamically allocated; and the first context feature and the second context feature are dynamically fused according to the first fusion parameter and the second fusion parameter.

[0102] Among them, the Transformer can model dependencies at arbitrary distances, but it is not very sensitive to position and orientation information; the Bi-LSTM can capture orientation information well, but it cannot capture global information. An attention mechanism is used to dynamically fuse the contextual features extracted by the Transformer encoder and the Bi-LSTM, achieving complementary advantages. The dynamic fusion of attention mechanisms is implemented as follows:

[0103]

[0104]

[0105] W Z Let x be the learnable weight matrix, and σ be the α-sqrt activation function;t It is the vector output by the Transformer encoder, x b This is the vector output by the BiLSTM. The dimension of vector z is related to x. t and x b The same dimension is used for the weights between two vectors, which allows the model to dynamically determine how much information to use from the Transformer encoder or Bi-LSTM, thereby remembering important information and avoiding information redundancy.

[0106] Step S205: Decode the fused features to obtain the Chinese named entities corresponding to the text to be identified.

[0107] Following this embodiment, a Conditional Random Field (CRF) is used to decode the fused features and output the corresponding labels. To utilize the dependencies between different labels, this embodiment uses a CRF as the decoding layer. For a given sequence... The corresponding label sequence is Let Y(s) represent all valid label sequences, and the probability of y is calculated by the following equation.

[0108]

[0109] in It is a calculation from arrive State transition scores and The score, its goal During decoding, the Viterbi algorithm is used to find the globally optimal sequence.

[0110] This application proposes an attention-based Chinese named entity recognition method. Considering that the original Transformer encoder possesses global modeling capabilities but is insensitive to positional and directional information, the method improves the Transformer encoder by using relative positional encoding and modifying the attention calculation formula. Simultaneously, BiLSTM is utilized to further enhance directional information. Finally, an attention mechanism is used to dynamically fuse contextual features, achieving deep fusion of global semantic and directional information. To obtain more contextual information and address the polysemy problem, a RoBERTa-wwm pre-trained model is used as a character-level embedding, improving the model's recognition performance. Experimental results on the Resume and Weibo datasets demonstrate that, without using external resources such as dictionaries, the model in this embodiment still achieves better results than current mainstream deep learning models, proving the effectiveness of the proposed method in Chinese named entity recognition tasks.

[0111] In this embodiment, the Weibo dataset and the Resume dataset are used, both of which are publicly available Chinese named entity recognition datasets. The Resume dataset is a dataset of resumes of executives of Chinese stock market listed companies collected and annotated from Sina Finance, including eight named entities. The Weibo dataset is constructed based on text from Sina Weibo and contains four entities. The datasets used in the experiment were annotated using the BMESO (Begin, Middle, End, Single, Outside) method. Detailed statistics and partitioning of the two datasets are as follows: Figure 4 As shown.

[0112] The experiment used precision, recall, and F1 score as evaluation metrics. The model was built using the PyTorch 1.10 framework and accelerated using an NVIDIA GeForce RTX 2080 Ti GPU. Its main hyperparameters are as follows: Figure 5 As shown.

[0113] For ease of description, TBAC is used here to represent the Transformer-BiLSTM-Attention-CRF model proposed in this embodiment. To verify the effectiveness of the proposed method, we compared it with other Chinese named entity recognition methods on the Weibo and Resume datasets, including methods using Word2Vector word vectors and methods using pre-trained models. Ablation experiments were also conducted to verify the necessity of each part of the model. A brief description of the comparison models is as follows.

[0114] (1) Bi-LSTM: Using Bi-LSTM network as encoder and CRF as decoder, it is the most classic named entity recognition model.

[0115] (2) ID-CNN: It uses dilated convolutional neural networks instead of Bi-LSTM networks as encoders, making full use of the parallelism of GPUs to improve computational efficiency.

[0116] (3) Transformer: Use the original Transformer architecture as the encoder and CRF as the decoder.

[0117] (4) Lattice LSTM: LSTM is used to model word graphs in the context encoding layer, explicitly fusing lexical information, and finally using CRF as the decoder.

[0118] (5) CAN-NER: Uses a convolutional neural network with a local attention layer and a bidirectional gated recursive unit (Bi-GRU) with a global self-attention layer as the encoder.

[0119] (6) FLAT: All potential words are incorporated into the sequence input using a relative span encoding method, and then the Transformer encoder is used to model the interaction between the symbol and the words to introduce lexical information.

[0120] (7) MECT: This method uses character, word, and Chinese character structural information as multi-element embeddings and employs a Cross-Transformer as the encoder. Comparative analysis based on Word2Vector word vectors.

[0121] Comparative experiments were conducted using five models: Bi-LSTM, ID-CNN, Transformer, Lattice LSTM, and CAN-NER. All five models were pre-trained on a Chinese corpus using the Word2Vector method, as described in the literature, to obtain character-level embeddings and bigram character-level embeddings. Lattice LSTM used an external dictionary for vocabulary augmentation, while CAN-NER utilized word segmentation information. The experimental data from the original papers for Lattice LSTM and CAN-NER were used for comparison.

[0122] The Transformer's F1 scores on the Resume and Weibo datasets were 92.89% and 47.66%, respectively, both lower than Bi-LSTM and ID-CNN models. This indicates that the original Transformer encoder is not well-suited for named entity tasks. It is necessary to improve the Transformer encoder using methods such as relative positioning to leverage its advantage of perceiving global information.

[0123] as follows Figure 6 The results shown are from a comparative experiment based on Word2Vector word vectors.

[0124] Analysis of the data in the table shows that the method provided in this application achieved F1 scores of 95.26% and 60.24% on the Resume and Weibo datasets, respectively, both outperforming other comparative models. On the Resume dataset, the F1 score of the model proposed in this embodiment is 0.78 and 1.12 higher than the classic Bi-LSTM and ID-CNN models, respectively; and 0.8 and 0.32 higher than LatticeLSTM and CAN-NER, respectively, which use lexical information. On the Weibo dataset, the F1 score of the model proposed in this embodiment is 4.12 and 4.98 higher than the classic Bi-LSTM and ID-CNN models, respectively; and 1.45 and 0.93 higher than Lattice LSTM and CAN-NER, respectively, which use lexical information. The experimental results demonstrate that the model proposed in this embodiment can capture more contextual feature information, effectively identify entities in Chinese text, and is competitive with methods that use lexical information.

[0125] The experimental results show that the F1 scores of the various models on the Weibo dataset differ more significantly from those on the Resume dataset. This phenomenon may be due to the fact that the Weibo dataset uses text from the social media domain, which contains a large number of irregular grammars and new words, making recognition more difficult. Figure 6 It can be seen that the various models have high precision but low recall on the Weibo dataset, which is also due to the high difficulty of recognition on the Weibo dataset.

[0126] Based on comparative analysis of pre-trained models, this application uses four models—FLAT, MECT, BiLSTM, and ID-CNN—for comparative experiments. The experimental data is from the original paper. FLAT and MECT are fine-tuned using the BERT-wwm pre-trained model. BiLSTM and ID-CNN are fine-tuned using the RoBERTa-wwm pre-trained model. The proposed model TBAC in this embodiment is compared using both the BERT-wwm and RoBERTa-wwm pre-trained models. Figure 7 The results of the comparative experiment based on the pre-trained model are shown.

[0127] As above Figure 7 As shown, when using the BERT-wwm pre-trained model, the model proposed in this embodiment achieves an F1 score 2.09 and 0.21 higher than the FLAT and MECT models on the Weibo dataset, and an F1 score 0.53 and 0.41 higher than the FLAT and MECT models on the Resume dataset. Experimental results demonstrate that our proposed model remains competitive with these vocabulary-enhanced methods.

[0128] The TBAC model, pre-trained with RoBERTa-wwm, achieves F1 scores of 71.29% and 96.68% on the Weibo and Resume datasets, respectively, outperforming the model pre-trained with BERT-wwm. This is because the RoBERTa pre-trained model employs dynamic masking and Chinese full-word masking strategies, and utilizes more text for pre-training, thus acquiring more vocabulary and contextual information. It also achieves better recognition results compared to classic BiLSTM and ID-CNN models, demonstrating that our proposed TBAC model can extract more contextual semantic features.

[0129] To verify the effectiveness of each part of the model proposed in this application, ablation experiments were conducted. The results of the ablation experiments are as follows: Figure 8 As shown in the diagram. —BiLSTM replaces the Bi-LSTM network with a Transformer encoder, —Transformer indicates that the Transformer network is replaced with a Bi-LSTM network, and —Attention represents that direct concatenation is used instead of Attention for feature fusion.

[0130] As above Figure 8 As shown, without the Bi-LSTM network, the F1 score is reduced by 0.82 and 3.45 on the Resume and Weibo datasets, respectively. This indicates that the contextual features and directional information extracted by the Bi-LSTM network are important for named entity recognition tasks. Replacing the improved Transformer encoder with the Bi-LSTM network reduces the F1 score by 0.64 and 5.86 on the Resume and Weibo datasets, respectively. This demonstrates that the global semantic information brought by the improved Transformer encoder can improve the F1 score. Without using the attention mechanism to dynamically fuse features, the F1 score is reduced by 0.53 and 2.07 on the Resume and Weibo datasets, respectively. The experiments show that the attention mechanism effectively fuses directional and global semantic information and discards some redundant information, thus improving the model's recognition performance.

[0131] This application also provides an electronic device, such as... Figure 9 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically:

[0132] The electronic device may include components such as a processor 501 with one or more processing cores, a memory 502 with one or more computer-readable storage media, a power supply 503, and an input unit 504. Those skilled in the art will understand that the electronic device structure shown in the figures does not constitute a limitation on the electronic device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0133] The processor 501 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 502, and by calling data stored in the memory 502, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. Optionally, the processor 501 may include one or more processing cores; preferably, the processor 501 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 501.

[0134] The memory 502 can be used to store software programs and modules. The processor 501 executes various functional applications and data processing by running the software programs and modules stored in the memory 502. The memory 502 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 502 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 502 may also include a memory controller to provide the processor 501 with access to the memory 502.

[0135] The electronic device also includes a power supply 503 that supplies power to various components. Preferably, the power supply 503 can be logically connected to the processor 501 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 503 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0136] The electronic device may also include an input unit 504, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0137] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 501 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 502 according to the following instructions, and the processor 501 runs the applications stored in the memory 502 to realize various functions, as follows: inputting the text to be recognized into the embedding layer to obtain word vectors; using a Transformer encoder to extract features from the word vectors to obtain first context features; using a Bi-LSTM model to extract features from the word vectors to obtain second context features; fusing the first context features and the second context features to obtain fused features; and decoding the fused features to obtain the Chinese named entities corresponding to the text to be recognized.

[0138] Furthermore, the step of using a Transformer encoder to extract features from the word vectors to obtain the first contextual features includes:

[0139] The word vectors are analyzed using a Transformer encoder to obtain the relative position encoding of each character;

[0140] The attention scores of the word vectors and the relative position encodings are calculated respectively to obtain the first contextual features.

[0141] Furthermore, the second contextual features obtained by using the Bi-LSTM model to extract features from the word vectors include:

[0142] The Bi-LSTM model is used to input the word vectors with forward and backward directions respectively. The forward and backward vectors are calculated and then concatenated to obtain the second contextual feature.

[0143] Furthermore, the fusion of the first context feature and the second context feature to obtain the fused feature includes:

[0144] Dynamically allocate the first fusion parameter of the first context feature and the second fusion parameter of the second context feature;

[0145] The first context feature and the second context feature are dynamically fused according to the first fusion parameter and the second fusion parameter.

[0146] Furthermore, decoding the fused features includes:

[0147] The fused features are decoded using a conditional random field, and corresponding labels are output.

[0148] Furthermore, the step of inputting the text to be recognized into the embedding layer to obtain word vectors includes:

[0149] The text to be identified is analyzed to obtain character-level embedding and binary character-level word embedding. Character-level embedding is based on characters, while binary character-level word embedding is based on pairs of characters.

[0150] Word vectors are obtained by concatenating character-level embeddings and bigram character-level word embeddings.

[0151] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0152] To this end, embodiments of this application provide a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk, or optical disk, etc. A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in the method provided in embodiments of this application. For example, the computer program loaded by the processor may execute the following steps: inputting the text to be recognized into an embedding layer to obtain word vectors; using a Transformer encoder to extract features from the word vectors to obtain first context features; using a Bi-LSTM model to extract features from the word vectors to obtain second context features; fusing the first context features and the second context features to obtain fused features; and decoding the fused features to obtain the Chinese named entities corresponding to the text to be recognized.

[0153] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the detailed descriptions of other embodiments above, which will not be repeated here.

[0154] In practice, each of the above units or structures can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units or structures, please refer to the previous method embodiments, which will not be repeated here.

[0155] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0156] The above describes a Chinese named entity recognition method, electronic device, and storage medium based on an attention mechanism provided in the embodiments of this application. Specific examples have been used in this application to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A Chinese named entity recognition method based on an attention mechanism, characterized in that, include: The text to be identified is input into the embedding layer to obtain word vectors. Specifically, the text to be identified is analyzed to obtain character-level embeddings and bigram character-level word embeddings. Character-level embeddings are based on characters, while bigram character-level word embeddings are based on pairs of characters. The character-level embeddings and bigram character-level word embeddings are concatenated to obtain word vectors. The relative position encoding of each character is obtained by using a preset calculation formula. The Transformer encoder is used to analyze the word vector and the relative position encoding to obtain an attention score, thereby obtaining the first context feature. The Transformer encoder calculates the word vector and the relative position encoding separately when calculating the attention score, and adds bias terms for relative distance and direction. The word vectors are used to extract features to obtain the second context features; An attention score is obtained by employing an attention mechanism. The attention score is then used as a fusion parameter to dynamically allocate a first fusion parameter for the first context feature and a second fusion parameter for the second context feature. The first context feature and the second context feature are then dynamically fused according to the first fusion parameter and the second fusion parameter to obtain a fused feature. The fused features are decoded to obtain the Chinese named entities corresponding to the text to be identified.

2. The Chinese named entity recognition method as described in claim 1, characterized in that, The second context features obtained by using the Bi-LSTM model to extract features from the word vectors include: The Bi-LSTM model is used to input the word vectors with forward and backward directions respectively. The forward and backward vectors are calculated and then concatenated to obtain the second contextual feature.

3. The Chinese named entity recognition method as described in claim 1, characterized in that, Decoding the fused features includes: The fused features are decoded using a conditional random field, and corresponding labels are output.

4. The Chinese named entity recognition method as described in claim 1, characterized in that, Word2Vector word vectors and a pre-trained model are used as embedding layers.

5. An electronic device, characterized in that, The electronic device includes: One or more processors; Memory; and One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the method of any one of claims 1 to 4.

6. A computer-readable storage medium, characterized in that, It contains a computer program that is loaded by a processor to perform the steps of the method according to any one of claims 1 to 4.

Images