A named entity recognition method, system, storage medium and terminal

Abstract

The application discloses a named entity recognition method and system, a storage medium and a terminal, and comprises the following steps: using multiple masking methods to mask tokens of words in a sentence, and training a BERT-Convolution model; converting the tokens into multiple token embeddings, and adding the multiple token embeddings as inputs of the BERT-Convolution model; using the BERT-Convolution model to perform word vector coding on each token according to context, wherein the token embedding is dynamically convolved, and the word vector features obtained through convolution are spliced with results obtained through a self-attention mechanism; the BERT-Convolution model outputs N*D word vector information, and then the word vector information is subjected to vector transformation through a full connection layer to output a score vector of each entity label; a score matrix formed by the score vector is input into a CRF layer, a Viterbi algorithm is used for decoding, and a path of an entity label with maximum probability is found. The application can improve the attention of the self-attention mechanism in BERT to local information, more accurately extract entities, and optimize the calculation complexity in the self-attention mechanism.

Description

Technical Field

[0001] This invention relates to the field of entity recognition technology, and in particular to a named entity recognition method, system, storage medium, and terminal. Background Technology

[0002] Named Entity Recognition (NER) is a subtask in Natural Language Processing (NLP) that aims to extract entities from unstructured text and determine their category, such as public domain entity tags: person names, place names, and organization names. NER serves not only as an independent tool for information extraction but also plays a crucial role in various NLP applications, such as knowledge graph construction and question-answering systems. Methods for implementing NER include rule-based, unsupervised learning, supervised learning using human-defined features, and deep learning-supervised learning.

[0003] Supervised deep learning typically consists of three modules: Embedding, which converts each token into a word vector (a representative method is Word2Vec, with models like CBOW and Skip-gram); Context Encoder, which extracts hidden information from the context of a sentence (representative models include BILSTM, GPT, and BERT); and Label Decoder, the final part of the NER model, which takes the representation obtained from the context encoder as input (common decoders include MLP+SoftMax and CRF).

[0004] Before using pre-trained models, the commonly used entity extraction method was BILSTM+CRF, which required pre-training word vectors using methods such as Word2Vec. Using pre-trained BERT allows learning a language model from a large-scale corpus first, and then fine-tuning it for downstream tasks. It eliminates the need for pre-trained word vectors and effectively utilizes information from the pre-training corpus, making BERT crucial in downstream tasks. With the addition of BERT, the contextual information captured by BILSTM is already implicit in BERT, and since BiLSTM cannot be parallelized, this patent's baseline entity extraction method is BERT+CRF to improve entity extraction efficiency. BERT is a model that uses the Transformer Encoder part for encoding. The core of the Transformer Encoder is feature extraction using a self-attention mechanism. While self-attention can effectively extract global features, it has two drawbacks: 1. The original BERT is insensitive to local information. Because BERT's pre-training uses random masking and its internal Self-attention mechanism establishes a fully connected graph, the original BERT can only focus on global information; secondly, the BERT+BILSTM+CRF architecture has high space and computational complexity, making efficient computation impossible. Firstly, the computational complexity of BERT's internal dot product similarity is... Where N is the sequence length. Secondly, BiLSTM cannot be computed in parallel, and this complexity makes it difficult to handle long sequences. Summary of the Invention

[0005] The purpose of this invention is to overcome the problems of insensitivity to local information and high computational complexity in existing entity recognition methods, and to provide a named entity recognition method, system, storage medium and terminal.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] In the first approach, a named entity recognition method is provided, the method comprising the following steps:

[0008] S1. Input the sentence to be recognized, use various masking methods to mask the tokens of words in the sentence, and train the BERT-Convolution model;

[0009] S2. Perform word segmentation preprocessing on sentences with a length of N tokens to obtain word tokens that conform to the vocabulary. Convert the tokens into multiple token embeddings and add the multiple token embeddings together as the input to the BERT-Convolution model, where N is the sentence length.

[0010] S3. Use the BERT-Convolution model to encode word vectors for each token according to the context. The BERT-Convolution model integrates dynamic convolution and self-attention mechanism. The token embedding is dynamically convolved and the word vector features obtained by the convolution are concatenated with the results obtained by the self-attention mechanism.

[0011] S4. The BERT-Convolution model outputs N*d word vector information, which is then transformed through a fully connected layer to output the score vector of each entity label, where d is the dimension of the embedding.

[0012] S5. Input the fraction matrix composed of fraction vectors into the CRF layer, use the Viterbi algorithm to decode it, find the entity label path with the highest probability, and identify the entity in the sentence based on the entity label path with the highest probability.

[0013] As a preferred option, a named entity recognition method includes a basic-level masking strategy, an entity-level masking strategy, and a phrase-level masking strategy.

[0014] As a preferred option, a named entity recognition method, wherein multiple masking methods are used to mask the tokens of words in a sentence, including:

[0015] Basic-Level Masking is used to randomly mask 15% of basic language units. For words that are removed from the original sentence, 80% of the words are replaced with a special symbol [MASK], 10% of the words are replaced with an arbitrary word, and the remaining 10% of the words remain unchanged. Other tokens in the sentence are used as input to predict the masked tokens.

[0016] The new Entity-Level Masking feature masks all the tokens of entities that were only partially masked in Basic-Level Masking, allowing the model to predict the entire entity.

[0017] Using Phrase-Level Masking, several phrases are randomly selected from a sentence, masked, and all basic units within the same phrase are predicted.

[0018] As a preferred option, a named entity recognition method includes the following word segmentation preprocessing:

[0019] The input Chinese characters are segmented into individual characters, while the English characters are segmented using the BertTokenize tokenizer.

[0020] As a preferred option, a named entity recognition method, wherein converting a token into multiple token embeddings and summing the multiple token embeddings as input to the BERT-Convolution model, includes:

[0021] The preprocessed sentences are used to generate a vocabulary vector library, an NSP vector library, and a position vector library, respectively. The vocabulary vector library, NSP vector library, and position vector library are then added together and used as the input to the BERT-Convolution model.

[0022] As a preferred option, a named entity recognition method, wherein the BERT-Convolution model integrates dynamic convolution and self-attention mechanisms, including:

[0023] The dynamic convolution and self-attention mechanism are combined to form a hybrid attention block. The embedded features are projected into a low-dimensional space through a linear transformation and then processed by the self-attention mechanism.

[0024] As a preferred option, a named entity recognition method, wherein the dynamic convolution includes dynamic depthwise separable convolution and lightweight convolution.

[0025] In the second approach, a named entity recognition system is provided, the system comprising:

[0026] The BERT-Convolution model pre-training module is used to take a sentence to be recognized as input, mask the tokens of words in the sentence using various masking methods, and train the BERT-Convolution model.

[0027] The BERT-Convolution model is used to preprocess sentences of length N tokens by word segmentation to obtain word tokens that conform to the vocabulary. These tokens are then converted into multiple token embeddings, and the sum of these embeddings is used as the input to the BERT-Convolution model, where N is the sentence length. The BERT-Convolution model encodes word vectors for each token based on the context. It integrates dynamic convolution and self-attention mechanisms, performing dynamic convolution on the token embeddings and concatenating the resulting word vector features with the results from the self-attention mechanism. The BERT-Convolution model outputs N*d word vector information, which is then transformed through a fully connected layer to output a score vector for each entity label, where d is the embedding dimension.

[0028] The CRF module is used to input a fraction matrix composed of fraction vectors into the CRF layer, use the Viterbi algorithm to decode it, find the entity label path with the highest probability, and identify the entity in the sentence based on the entity label path with the highest probability.

[0029] In the third approach, a storage medium is provided that stores computer instructions, which, when executed, perform any of the named entity recognition methods described above.

[0030] In the fourth approach, a terminal is provided, including a memory and a processor. The memory stores computer instructions that can be executed on the processor. When the processor executes the computer instructions, it performs any of the named entity recognition methods described above.

[0031] It should be further noted that the technical features corresponding to the above options can be combined or substituted to form new technical solutions if there is no conflict.

[0032] Compared with the prior art, the beneficial effects of the present invention are:

[0033] (1) The BERT-Convolution model of the present invention integrates dynamic convolution and self-attention mechanism. The token embedding is dynamically convolved and the word vector features obtained by convolution are concatenated with the results obtained by self-attention mechanism. Local information is embedded into global information, which can better extract the local information of the entity in the input sentence. It can improve the attention of the self-attention mechanism in BERT to local information and extract entities more accurately. At the same time, the combination of dynamic convolution and self-attention mechanism reduces the number of redundant heads, optimizes the computational complexity in self-attention mechanism, and saves a lot of computational cost of self-attention.

[0034] (2) In one example, in addition to BERT's basic token-level masking strategy, entity masking and phrase-level masking strategies were added during the training phase. This masked several words belonging to the same entity category or the same phrase, instead of just masking a single word or character. In this way, prior knowledge of phrases and entities is implicitly learned during training. This allows the model to implicitly learn information about knowledge and longer semantic dependencies, such as relationships between entities and entity attributes, to guide word embedding learning, thereby making the model more general and adaptable. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating a named entity recognition method according to an embodiment of the present invention;

[0036] Figure 2 This is a schematic diagram illustrating the process of word segmentation using three masking methods and multiple token embeddings combined, as shown in an embodiment of the present invention.

[0037] Figure 3 This is a schematic diagram of the overall structure of the BERT-Convolution model as shown in an embodiment of the present invention;

[0038] Figure 4 This is a schematic diagram of pre-training as shown in an embodiment of the present invention;

[0039] Figure 5 This is a schematic diagram illustrating word embedding in an embodiment of the present invention;

[0040] Figure 6 This is a schematic diagram of BERT-Convolution as shown in an embodiment of the present invention;

[0041] Figure 7 This is a schematic diagram of the Viterbi algorithm's annotation process for a sentence, as shown in an embodiment of the present invention. Detailed Implementation

[0042] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0044] To better describe the embodiments of the present invention, the following terms are explained:

[0045] Token: In computer identity authentication, it means a token (temporary); in lexical analysis, it means a mark.

[0046] Embedding: Embedding, also called mapping, is the process of mapping a sentence composed of words to a representation vector.

[0047] Mask: Masking method; Basic-Level Masking: Basic token-level masking strategy; Entity-Level Masking: Entity masking strategy; Phrase-Level Masking: Phrase-level masking strategy.

[0048] BERT stands for Bidirectional Encoder Representations from Transformer. It's a pre-trained language representation model that departs from traditional unidirectional language models or shallow concatenation of two unidirectional models for pre-training. Instead, it employs a novel masked language model (MLM) to generate deep bidirectional language representations. The goal of BERT is to train on large-scale unlabeled corpora to obtain semantically rich representations of text, and then fine-tune these representations for specific NLP tasks before applying them to those tasks.

[0049] CRF: Conditional Random Field is a fundamental model in natural language processing, widely used in scenarios such as word segmentation, entity recognition, and part-of-speech tagging.

[0050] Viterbi: The Viterbi algorithm is a dynamic programming algorithm used to find the Viterbi path—the sequence of hidden states—most likely to produce a sequence of observed events, particularly in the context of Markov information sources and Hidden Markov Models. The terms "Viterbi path" and "Viterbi algorithm" are also used to describe dynamic programming algorithms that find the most likely explanations for observations. For example, in statistical syntactic analysis, dynamic programming algorithms can be used to find the most likely context-independent derived (parsed) strings, sometimes referred to as "Viterbi analysis."

[0051] In one exemplary embodiment, a named entity recognition method is provided, such as Figure 1 As shown, the method includes the following steps:

[0052] S1. Input the sentence to be recognized, use various masking methods to mask the tokens of words in the sentence, and train the BERT-Convolution model;

[0053] S2. Perform word segmentation preprocessing on sentences with a length of N (512) tokens to obtain word tokens that match the vocabulary. Convert the tokens into multiple token embeddings and add the multiple token embeddings as the input to the BERT-Convolution model, where N is the sentence length.

[0054] S3. Use the BERT-Convolution model to encode word vectors for each token according to the context. The BERT-Convolution model integrates dynamic convolution and self-attention mechanism. The token embedding is dynamically convolved and the word vector features obtained by the convolution are concatenated with the results obtained by the self-attention mechanism.

[0055] The S4 and BERT-Convolution models output word vector information in 512*768 dimensions. Then, the vectors are transformed through a fully connected layer to output the score vector of each entity label. If there are 5 types of entity labels (B-Target1), then 512 5-dimensional score vectors will be output.

[0056] S5. Input the fraction matrix composed of fraction vectors into the CRF layer, use the Viterbi algorithm to decode it, find the entity label path with the highest probability, and identify the entity in the sentence based on the entity label path with the highest probability.

[0057] Specifically, a named entity recognition method, such as Figure 2 , Figure 4As shown, the masking method includes Basic-Level Masking, Entity-Level Masking, and Phrase-Level Masking. Specifically, S1 includes:

[0058] First, Basic-Level Masking is used, which treats a sentence as a sequence of tokens representing basic language units. For English, the basic language unit is a word, and for Chinese, it is a character. During training, 15% of the basic language units are randomly masked. For the words that are removed from the original sentence, 80% of the words are replaced with a special symbol [MASK], 10% are replaced with an arbitrary word, and the remaining 10% remain unchanged. Other tokens in the sentence are used as input to predict the masked tokens.

[0059] Named entities include names of people, places, organizations, products, etc. They can be abstract or physically existing, and typically contain important information within a sentence. After using Basic-Level Masking, a new Entity-Level Masking method is used to analyze named entities in the sentence, then mask and predict all slots within the entities. This masks all tokens of entities that were only partially masked in Basic-Level Masking, allowing the model to predict the entire entity.

[0060] Finally, Phrase-Level Masking is used to randomly select several phrases in the sentence, mask them, and predict all basic units within the same phrase. Specifically, a phrase is a group of words or characters that function as conceptual units. For English, lexical analysis and chunking tools are used to obtain the boundaries of phrases in the sentence, and some language-related segmentation tools are used to obtain word / phrase information for other languages ​​(such as Chinese).

[0061] In one example, a named entity recognition method includes word segmentation preprocessing comprising:

[0062] The input sentence is preprocessed by word segmentation. Chinese is generally segmented directly into individual characters, while English is segmented using the BertTokenize word segmenter.

[0063] Furthermore, a named entity recognition method, wherein the token is converted into multiple token embeddings and the multiple token embeddings are summed as input to the BERT-Convolution model, includes:

[0064] The preprocessed sentences are used to generate a vocabulary vector library, NSP vectors, and a position vector library, respectively. These three libraries are then summed and used as input to the BERT-Convolution model. Specifically, as follows... Figure 2 , Figure 5 As shown, S2 specifically includes:

[0065] The preprocessed sentences are first used to generate a word vector library with a size of [21128, 768], where 21128 is the size of the Chinese word list and each word vector has a dimension of 768. The word indices Input_ids are then converted into word vectors Word_Embeddings.

[0066] Then, an NSP vector library is generated with a size of [2, 768], meaning it only represents 0 and 1 tags, and each word vector has a dimension of 768. The context relationship index Token_Type_ids is then converted into context relationship vector Token_Type_Embeddings.

[0067] Next, position indices `Position_ids` are generated sequentially, with indices ranging from 0 to 512. A position vector library is generated, with a size of [512, 768], meaning a length of 512 and a dimension of 768 for each word vector. The position indices `Position_ids` are then converted into position vectors `Position_Embeddings`. The three embeddings are summed and used as input to the pre-trained model.

[0068] In one example, a named entity recognition method, wherein the BERT-Convolution model integrates dynamic convolution and self-attention mechanisms, includes:

[0069] A hybrid attention block is formed by combining dynamic convolution and self-attention mechanisms. Embedded features are projected onto a low-dimensional space through a linear transformation and then processed by the self-attention mechanism. The dynamic convolution includes dynamically depthwise separable convolution and lightweight convolution. Now, combining... Figure 3 , Figure 6 Explain the principle behind the integration of dynamic convolution and self-attention mechanisms in the BERT-Convolution model.

[0070] Specifically, the hybrid attention block of the BERT-Convolution model integrates dynamic convolution and self-attention mechanisms. Some attention heads are redundant, thus reducing the number of attention heads while introducing dynamic convolution. The input embedding is first projected into a lower-dimensional space, and then through the self-attention module in the original BERT, the embedding features of dimension d are projected into the original transformer architecture with the query, key, and value of the same dimension d through a linear transformation. Conversely, we project the embedding features into a smaller space of dimension d / Y for further processing, where Y > 1 is the reduction ratio. Simultaneously, the number of attention heads is reduced by the ratio Y. This significantly saves computational costs.

[0071] Furthermore, the hybrid attention formula is expressed as:

[0072] MIX-Attetion(X, W) f ,I)=CatAttention(Q,K,V),DConv(X,W f ,I)

[0073] Cat() represents the connection between the Self-attention mechanism and the dynamic convolution, Q, K, and V are the attention calculation matrices, and Dconv() is the dynamic convolution. See formula (3) for details. Self-attention and dynamic convolution share the same Q matrix, but use different K matrices to generate attention maps and convolution kernels.

[0074] The formula for self-attention is:

[0075]

[0076] X is the input word vector, X∈R (n×d) The input vector has a length of n and a vector dimension of d. The input vector is transformed into Q, K, V∈R through three corresponding matrices. (n×d) Three attention vectors. The matrix Q... T ×K, after normalization and softmax, output the weight matrix, and then perform a weighted sum with V to obtain the attention vector.

[0077] The formulas for lightweight and dynamic convolution are as follows:

[0078]

[0079] The convolution kernel is represented as W∈R (n×d)d represents the dimension of word embedding, i represents the position, c represents the channel, j is the kernel number, k is the window size of a standard CNN, h represents the segmented sub-channel, and softmax() represents the normalization of parameters and dropout operation on each segmented sub-channel.

[0080] The formula for dynamically depthwise separable convolution is as follows:

[0081] DConv(X, W) f ,I)=LConvX,softmax(W f X i ), i (3)

[0082] Wherein, the convolution kernel is represented as W∈R (n×d) The output of the depthwise convolution at position i and channel c can be formulated as a light convolution, W f The convolution kernel is dynamically generated based on position i. The dynamic convolution is represented by the dynamic depthwise separable convolution formula.

[0083] This invention introduces convolution operations into the TokenEmbedding while performing Token Embedding extraction in BERT, convolving N*d features. Here, dynamically depthwise separable convolutions with shared parameters and lightweight convolutions are introduced to better extract local entity information from the input sentence. The word vector features obtained from the convolutions are concatenated with those obtained through a self-attention mechanism, embedding local information into global information. Furthermore, BERT uses multi-head self-attention; for example, a 12-head self-attention system has redundant heads. By introducing a dynamic convolution-based module, the number of heads is reduced. This significantly saves computational costs for self-attention and forces the attention heads to generate more compact and useful attention information.

[0084] In one example, the Viterbi algorithm can be summarized in the following three points:

[0085] 1. If the path with the highest probability passes through a point in the network, then the sub-path from the starting point to that point must also be the path with the highest probability from the starting point to that point.

[0086] 2. Assume there are k states at time i, and there are k shortest paths from the beginning to the k states at time i. The final shortest path must pass through one of these paths.

[0087] 3. According to the above property, when calculating the shortest path of the (i + 1)-th state, it is only necessary to consider the shortest path of the k state values from the start to the current state and the shortest path from the current state value to the (i + 1)-th state value. For example, when calculating the shortest path at t = 3, it is equal to the sum of the shortest paths of all state nodes x2i at t = 2 plus the shortest paths of each node from t = 2 to t = 3.

[0088] The following is the annotation process of a sentence by the Viterbi algorithm. Using the dynamic programming algorithm, generally when calculating the optimal candidate connections of each layer, the sum of the probabilities of the previous connections is recorded, and the index of the corresponding state node is recorded (the reason why the Viterbi algorithm is called the dynamic programming algorithm is this way of recording the calculated results for subsequent use). In this way, when reaching the last layer, the connection with the largest probability among the candidate connections of the last layer is the connection on the optimal path. Then, backtracking from this connection to find the complete path is the optimal path. As Figure 7 shown, there are 3 candidate optimal paths. Check which path has the largest sum of probabilities, and that path is the optimal path. The final result is to label the two characters "Xiaohong" as BI, and the others as O.

[0089] In the second solution, a named entity recognition system is provided. The system includes:

[0090] A BERT-Convolution model pre-training module, which is used to input the sentence to be recognized, mask the tokens of the words in the sentence using various masking methods, and train the BERT-Convolution model;

[0091] The BERT-Convolution model. The BERT-Convolution model is used to perform word segmentation preprocessing on a sentence with a length limited to N tokens to obtain words tokens that conform to the vocabulary, convert the tokens into multiple token embeddings, and add the multiple token embeddings as the input of the BERT-Convolution model, where N is the sentence length; use the BERT-Convolution model to perform word vector encoding on each token according to the context. Among them, the BERT-Convolution model integrates a dynamic convolution and a Self-attention mechanism to perform dynamic convolution on the token embedding and splice the word vector features obtained by convolution with the result obtained by the self-attention mechanism; the output of the BERT-Convolution model is word vector information of N * d, and then vector transformation is performed through a fully connected layer to output a score vector for each entity label, where d is the dimension of the embedding;

[0092] The CRF module is used to input a fraction matrix composed of fraction vectors into the CRF layer, use the Viterbi algorithm to decode it, find the entity label path with the highest probability, and identify the entity in the sentence based on the entity label path with the highest probability.

[0093] In the third approach, a storage medium is provided that stores computer instructions, which, when executed, perform any of the named entity recognition methods described above.

[0094] Based on this understanding, the technical solution of this embodiment, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0095] In the fourth approach, a terminal is provided, including a memory and a processor. The memory stores computer instructions that can be executed on the processor. When the processor executes the computer instructions, it performs any of the named entity recognition methods described above.

[0096] The processor may be a single-core or multi-core central processing unit or a specific integrated circuit, or one or more integrated circuits configured to implement the present invention.

[0097] The embodiments of the subject matter and functional operation described in this specification can be implemented in: tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing device or for controlling the operation of a data processing device. Alternatively or additionally, the program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by the data processing device.

[0098] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flow can also be executed by dedicated logic circuitry—such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the device can also be implemented as dedicated logic circuitry.

[0099] Suitable processors for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0100] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0101] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0102] The above detailed embodiments are a description of the present invention. It should not be considered that the specific embodiments of the present invention are limited to these descriptions. For those skilled in the art, several simple deductions and substitutions can be made without departing from the concept of the present invention, and all of these should be considered to fall within the protection scope of the present invention.

Claims

1. A named entity recognition method, characterized in that, The method includes the following steps: S1. Input the sentence to be recognized, use various masking methods to mask the tokens of words in the sentence, and train the BERT-Convolution model; S2. Perform word segmentation preprocessing on sentences with a length limited to N tokens to obtain word tokens that conform to the vocabulary. Convert the tokens into multiple token embeddings and sum the multiple token embeddings as input to the BERT-Convolution model, where N is the sentence length. The step of converting tokens into multiple token embeddings and summing the multiple token embeddings as input to the BERT-Convolution model includes: A vocabulary vector library, an NSP vector library, and a position vector library are generated for the preprocessed sentences, and the vocabulary vector library, NSP vector library, and position vector library are added together as the input to the BERT-Convolution model; S3. Encode word vectors for each token using the BERT-Convolution model based on the context. The BERT-Convolution model integrates dynamic convolution and self-attention mechanisms, performing dynamic convolution on the token embedding and concatenating the word vector features obtained from the convolution with the results obtained from the self-attention mechanism. The BERT-Convolution model integrates dynamic convolution and self-attention mechanisms, including: Dynamic convolution and self-attention mechanism are combined to form a hybrid attention block. The embedded features are projected into a low-dimensional space through linear transformation and then processed by the self-attention mechanism. S4. The BERT-Convolution model outputs N*d word vector information, which is then transformed through a fully connected layer to output the score vector of each entity label, where d is the dimension of the embedding. S5. Input the fraction matrix composed of fraction vectors into the CRF layer, use the Viterbi algorithm to decode it, find the entity label path with the highest probability, and identify the entity in the sentence based on the entity label path with the highest probability.

2. The named entity recognition method according to claim 1, characterized in that, The masking methods include Basic-Level Masking, Entity-Level Masking, and Phrase-Level Masking.

3. The named entity recognition method according to claim 2, characterized in that, The method of masking tokens for words in a sentence using multiple masking techniques includes: Basic-Level Masking is used to randomly mask 15% of basic language units. For words that are removed from the original sentence, 80% of the words are replaced with a special symbol [MASK], 10% of the words are replaced with an arbitrary word, and the remaining 10% of the words remain unchanged. Other tokens in the sentence are used as input to predict the masked tokens. The new Entity-Level Masking feature masks all the tokens of entities that were only partially masked in Basic-Level Masking, allowing the model to predict the entire entity. Using Phrase-Level Masking, several phrases are randomly selected from a sentence, masked, and all basic units within the same phrase are predicted.

4. The named entity recognition method according to claim 1, characterized in that, The word segmentation preprocessing includes: The input Chinese characters are segmented into individual characters, while the English characters are segmented using the BertTokenize tokenizer.

5. The named entity recognition method according to claim 1, characterized in that, The dynamic convolution includes dynamic depthwise separable convolution and lightweight convolution.

6. A named entity recognition system, characterized in that, The system includes: The BERT-Convolution model pre-training module is used to take a sentence to be recognized as input, mask the tokens of words in the sentence using various masking methods, and train the BERT-Convolution model. The BERT-Convolution model is used to preprocess sentences of length N tokens to obtain word tokens that conform to the vocabulary. These tokens are then converted into multiple token embeddings, and the sum of these embeddings is used as input to the BERT-Convolution model, where N is the sentence length. The BERT-Convolution model encodes word vectors for each token based on the context. It integrates dynamic convolution and self-attention mechanisms, performing dynamic convolution on the token embeddings and concatenating the resulting word vector features with the self-attention mechanism. The BERT-Convolution model outputs N*d word vector information, which is then transformed through a fully connected layer to output a score vector for each entity label, where d is the embedding dimension. The process of converting tokens into multiple token embeddings and summing them as input to the BERT-Convolution model includes: A vocabulary vector library, an NSP vector library, and a position vector library are generated for the preprocessed sentences, and the vocabulary vector library, NSP vector library, and position vector library are added together as the input to the BERT-Convolution model; The BERT-Convolution model integrates dynamic convolution and self-attention mechanisms, including: Dynamic convolution and self-attention mechanism are combined to form a hybrid attention block. The embedded features are projected into a low-dimensional space through linear transformation and then processed by the self-attention mechanism. The CRF module is used to input a fraction matrix composed of fraction vectors into the CRF layer, use the Viterbi algorithm to decode it, find the entity label path with the highest probability, and identify the entity in the sentence based on the entity label path with the highest probability.

7. A storage medium storing computer instructions thereon, characterized in that, The computer instructions execute the named entity recognition method according to any one of claims 1-5.

8. A terminal, comprising a memory and a processor, wherein the memory stores computer instructions executable by the processor, characterized in that, The named entity recognition method according to any one of claims 1-5 is executed when the processor runs computer instructions.

Images