Named entity recognition method and device and product

Abstract

The invention relates to the technical field of natural language processing, and discloses a named entity recognition method and device and a product, and the method comprises the steps: obtaining a to-be-recognized text sequence; extracting embedding vectors of the text sequence, wherein the embedding vectors comprise a word embedding vector, a character embedding vector, a vocabulary embedding vector and a syntax embedding vector; fusing the word embedding vector, the character embedding vector and the vocabulary embedding vector to obtain a first fusion vector, and extracting a global feature tensor and a local feature tensor of the first fusion vector; performing feature enhancement on the syntactic embedding vector to obtain a deep grammar feature vector; and generating a named entity recognition result of the text sequence based on the global feature tensor, the local feature tensor and the deep grammar feature vector. According to the method, the global feature tensor and the local feature tensor are fused, the context information of the text can be effectively utilized, the deep grammar feature vector is added, the understanding of the grammar structure can be enhanced, and therefore the recognition precision of the named entity is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of natural language processing technology, and in particular to a method, device and product for named entity recognition. Background Art

[0002] Named entity recognition is a basic and key task in the field of natural language processing, which aims to identify and classify named entities with specific meanings from text, such as names of people, places, names of organizations, time, date, etc. As the cornerstone of many advanced natural language processing applications, efficient and accurate named entity recognition methods have always been a research hotspot in academia and industry.

[0003] In recent years, deep learning technology has made breakthrough progress in the field of natural language processing, greatly promoting the development of named entity recognition technology. Named entity recognition methods based on deep learning, such as recurrent neural networks, convolutional neural networks, and Transformer models, can automatically learn the feature representation of text, avoiding tedious manual feature engineering. However, these methods still face some challenges and limitations, resulting in relatively low recognition accuracy of named entities. Summary of the invention

[0004] In view of this, the present disclosure provides a method, device and product for named entity recognition to solve the problem of low recognition accuracy of named entities.

[0005] In a first aspect, the present disclosure provides a method for named entity recognition, the method comprising:

[0006] Get the text sequence to be recognized;

[0007] Extract the embedding vector of the text sequence, which includes word embedding vector, character embedding vector, vocabulary embedding vector and syntactic embedding vector;

[0008] Fusing the word embedding vector, the character embedding vector, and the vocabulary embedding vector to obtain a first fused vector, and extracting a global feature tensor and a local feature tensor of the first fused vector;

[0009] Perform feature enhancement on the syntactic embedding vector to obtain a deep grammatical feature vector;

[0010] Generate named entity recognition results for text sequences based on global feature tensors, local feature tensors, and deep grammatical feature vectors.

[0011] In an embodiment of the present disclosure, a text sequence to be recognized is obtained; an embedding vector of the text sequence is extracted, and the embedding vector includes a word embedding vector, a character embedding vector, a vocabulary embedding vector, and a syntax embedding vector; the word embedding vector, the character embedding vector, and the vocabulary embedding vector are fused to obtain a first fused vector, and a global feature tensor and a local feature tensor of the first fused vector are extracted; the syntax embedding vector is feature enhanced to obtain a deep grammatical feature vector; based on the global feature tensor, the local feature tensor, and the deep grammatical feature vector, a named entity recognition result of the text sequence is generated. Since the embodiment of the present disclosure fuses the global feature tensor and the local feature tensor, the contextual information of the text can be effectively utilized, and the deep grammatical feature vector is added to enhance the understanding of the grammatical structure, thereby improving the recognition accuracy of the named entity.

[0012] In an optional implementation, extracting the global feature tensor of the first fusion vector includes:

[0013] Based on the multi-head self-attention mechanism, the first fusion vector is converted into a first intermediate vector, and the first intermediate vector represents the dependency relationship between different positions in the text sequence;

[0014] Based on the first structured state space model, the first intermediate vector is converted into a second intermediate vector, where the second intermediate vector represents a global context representation of the text sequence;

[0015] Based on the feedforward neural network, the second intermediate vector is transformed to obtain the global feature tensor.

[0016] In the disclosed embodiment, by converting the first fusion vector into the first intermediate vector based on the multi-head self-attention mechanism, the dependency between any two positions in the text sequence can be captured, thereby effectively capturing long-distance context information, so that the model can better understand the relationship between each word in the sentence. By converting the first intermediate vector into the second intermediate vector based on the first structured state space model, global context modeling can be efficiently performed, and it has good parallelism, thereby improving computational efficiency. By converting the second intermediate vector based on a feedforward neural network to obtain a global feature tensor, the nonlinear expression ability of the model can be enhanced, and the global context information of the text can be obtained.

[0017] In an optional implementation, extracting a local feature tensor of the first fusion vector includes:

[0018] Based on a plurality of convolution kernels of different sizes, a convolution operation is performed on the first fusion vector to obtain a plurality of different feature maps;

[0019] Concatenate multiple different feature maps to obtain a local feature tensor.

[0020] In the disclosed embodiment, by performing a convolution operation on the first fusion vector to obtain a plurality of different feature maps, the local features of the text sequence can be effectively extracted, thereby capturing the local context information in the text sequence, which helps the model capture the local features inside the entity. By using convolution kernels of different sizes, local context information of different ranges can be captured, so that the model can better adapt to entities of different lengths and obtain the local context information of the text.

[0021] In an optional implementation, the text sequence is a text sequence in the target domain, and the syntactic embedding vector is feature enhanced to obtain a deep grammatical feature vector, including:

[0022] Encoding the syntactic embedding vector through a private bidirectional long short-term memory network to obtain a first encoding vector, wherein the first encoding vector includes grammatical feature information of the target domain;

[0023] The syntactic embedding vector is encoded by a shared bidirectional long short-term memory network to obtain a second encoding vector, wherein the second encoding vector includes grammatical feature information shared by multiple domains;

[0024] Based on the second structured state space model, converting the first encoding vector and the second encoding vector to obtain a third intermediate vector;

[0025] The first encoding vector, the second encoding vector and the third intermediate vector are fused to obtain a deep grammatical feature vector.

[0026] In the disclosed embodiment, the syntactic embedding vector is encoded by a private bidirectional long short-term memory network to obtain a first encoding vector, which can capture the dependency relationship between each word in the sentence and the words before and after it, and extract the grammatical features of the target domain. The syntactic embedding vector is encoded by a shared bidirectional long short-term memory network to obtain a second encoding vector, which can extract grammatical features shared across domains. By converting the first encoding vector and the second encoding vector based on the second structured state space model to obtain a third intermediate vector, the long-distance dependency relationship in the sequence can be efficiently captured. By fusing the first encoding vector, the second encoding vector and the third intermediate vector to obtain a deep grammatical feature vector, the syntactic information of the text can be obtained, the understanding of the grammatical structure of the text can be enhanced, and the accuracy of named entity recognition can be improved.

[0027] In an optional implementation, generating a named entity recognition result of a text sequence based on the global feature tensor, the local feature tensor, and the deep grammatical feature vector includes:

[0028] Concatenate the global feature tensor and the local feature tensor to obtain a first intermediate feature tensor;

[0029] Based on the attention weight matrix, the first intermediate feature tensor is weighted summed to obtain a fused feature tensor;

[0030] Concatenate the fused feature tensor and the deep syntax feature vector to obtain a second intermediate feature tensor;

[0031] Performing a nonlinear transformation on the second intermediate feature tensor to obtain a third intermediate feature tensor;

[0032] The third intermediate feature tensor is sequence labeled to obtain the named entity recognition result of the text sequence.

[0033] In the disclosed embodiment, by performing weighted summation on the first intermediate feature tensor based on the attention weight matrix to obtain a fused feature tensor, global features and local features can be effectively fused, thereby better utilizing the contextual information of the text. By performing a nonlinear transformation on the second intermediate feature tensor to obtain a third intermediate feature tensor, features can be further enhanced, more complex semantic relationships can be captured, and more discriminative features can be provided for sequence labeling tasks.

[0034] In an optional implementation, the first intermediate feature tensor includes sub-feature tensors of multiple time steps, and the attention weight matrix is ​​obtained based on the following method:

[0035] Based on the learned first weight matrix, feature mapping is performed on the first intermediate feature tensor to obtain a first mapped feature tensor;

[0036] Based on the learned second weight matrix, the first mapping feature tensor is linearly transformed to obtain the attention score of each sub-feature tensor;

[0037] The attention scores are averaged and normalized to obtain the attention weight matrix.

[0038] In the embodiment of the present disclosure, by performing feature mapping and linear transformation on the first intermediate feature tensor based on the learned first weight matrix and second weight matrix, and performing averaging and normalization to obtain the attention weight matrix, it is possible to adaptively learn the important relationship between global features and local features, and dynamically weight them according to their importance, ultimately obtaining a more expressive fused feature tensor.

[0039] In an optional implementation, performing a nonlinear transformation on the second intermediate feature tensor to obtain a third intermediate feature tensor includes:

[0040] Based on the B-spline basis function, the second intermediate feature tensor is mapped into a second mapped feature tensor, and based on the learned third weight matrix, the second mapped feature tensor is linearly transformed to obtain a first enhanced feature tensor;

[0041] Performing linear transformation and nonlinear transformation on the second intermediate feature tensor in sequence to obtain a second enhanced feature tensor;

[0042] The first enhanced feature tensor and the second enhanced feature tensor are added to obtain a third intermediate feature tensor.

[0043] In the embodiment of the present disclosure, by mapping and linearly transforming the second intermediate feature tensor based on the B-spline basis function and the learned third weight matrix, and further performing linear transformation and nonlinear transformation, it is possible to more flexibly fit complex nonlinear relationships, thereby improving the feature expression ability of the model and better capturing complex semantic relationships in the text.

[0044] In an optional implementation, sequence labeling is performed on the third intermediate feature tensor to obtain a named entity recognition result of the text sequence, including:

[0045] The third intermediate feature tensor is input into a plurality of different conditional random field components respectively, and the conditional random field components respectively determine the score of each label sequence based on the third intermediate feature tensor and the learned label transfer probability and emission probability, and the label sequence represents the candidate named entity recognition result;

[0046] Based on the scores of each label sequence obtained by each conditional random field component, determine the target label sequence with the highest score;

[0047] Based on the target label sequence, the named entity recognition result of the text sequence is obtained.

[0048] In the disclosed embodiment, by adopting multiple different conditional random field components for sequence labeling, the dependency relationship between different entity labels can be learned, and better labeling results can be obtained through global decoding, thereby improving the accuracy of named entity recognition.

[0049] In a second aspect, the present disclosure provides a named entity recognition device, the device comprising:

[0050] An acquisition module, used for acquiring a text sequence to be recognized;

[0051] A first extraction module is used to extract embedding vectors of text sequences, where the embedding vectors include word embedding vectors, character embedding vectors, vocabulary embedding vectors, and syntactic embedding vectors;

[0052] A second extraction module is used to fuse the word embedding vector, the character embedding vector and the vocabulary embedding vector to obtain a first fused vector, and extract a global feature tensor and a local feature tensor of the first fused vector;

[0053] The enhancement module is used to enhance the features of the syntactic embedding vector to obtain a deep grammatical feature vector;

[0054] The generation module is used to generate named entity recognition results of text sequences based on global feature tensors, local feature tensors and deep grammatical feature vectors.

[0055] In a third aspect, the present disclosure provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the named entity recognition method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0056] In a fourth aspect, the present disclosure provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method for named entity recognition of the first aspect or any corresponding embodiment thereof.

[0057] In a fifth aspect, the present disclosure provides a computer program product, including computer instructions, which are used to enable a computer to execute the named entity recognition method of the above-mentioned first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0059] Figure 1 is a network model architecture diagram of a named entity recognition method according to an embodiment of the present disclosure;

[0060] Figure 2 is a flowchart of a method for named entity recognition according to an embodiment of the present disclosure;

[0061] Figure 3 is a structural block diagram of a named entity recognition device according to an embodiment of the present disclosure;

[0062] Figure 4 It is a schematic diagram of the hardware structure of the computer device of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0064] Named entity recognition is a basic and key task in the field of natural language processing, which aims to identify and classify named entities with specific meanings from text, such as names of people, places, names of organizations, time, date, etc. As the cornerstone of many advanced natural language processing applications, efficient and accurate named entity recognition methods have always been a research hotspot in academia and industry.

[0065] In recent years, deep learning technology has made breakthrough progress in the field of natural language processing, greatly promoting the development of named entity recognition technology. Named entity recognition methods based on deep learning, such as recurrent neural networks, convolutional neural networks, and Transformer models, can automatically learn the feature representation of text, avoiding tedious manual feature engineering. However, these methods still face some challenges and limitations, resulting in relatively low recognition accuracy of named entities.

[0066] The related technologies still have the following problems when performing named entity recognition: First, the traditional word vector cannot fully express the multi-level semantic information of words, nor can it handle the problem of unregistered words well. Second, the existing technology cannot effectively fuse the local features and global features of the text. Third, most of the existing named entity recognition methods mainly focus on the semantic information of the text, while ignoring the grammatical information of the text, such as dependency syntactic information. In addition, due to its huge model parameters and computational complexity, the Transformer-based method is limited in its application on resource-constrained devices. Based on the above problems, the related technologies have problems such as insufficient feature representation capabilities, insufficient fusion of local information and global information, insufficient utilization of grammatical information, and high model parameters and computational complexity when performing named entity recognition.

[0067] In order to solve the above problems, according to an embodiment of the present disclosure, a method embodiment of named entity recognition is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0068] Before introducing the method disclosed in the present invention, a network model architecture is first introduced. Figure 1 As shown, Figure 1It is a network model architecture diagram of the named entity recognition method according to an embodiment of the present disclosure.

[0069] based on Figure 1 The network model architecture shown in the present disclosure provides a method for named entity recognition, such as Figure 2 As shown, Figure 2 : is a flowchart of a method for named entity recognition according to an embodiment of the present disclosure, and the flowchart can be applied to a server and includes the following steps:

[0070] Step S201, obtaining a text sequence to be recognized.

[0071] Optionally, in an embodiment of the present disclosure, the server obtains a text sequence to be recognized input by a user. Preferably, after obtaining the text sequence, the server can perform data preprocessing on the text sequence. For example, the server can annotate the text sequence in a BIO manner (B indicates the beginning of an entity, I indicates the inside of an entity, and O indicates that the current word does not belong to any entity), use the word segmenter provided by the pre-trained model to segment the text sequence, and then use a word unit conversion tool (such as the Tokenization tool provided in the Transformer library) to convert each word into a corresponding word unit sequence.

[0072] Specifically, since the deep learning model cannot directly process the original text, the server needs to decompose it into words or phrases first. The purpose of the word segmentation operation is to divide the continuous text sequence into independent words or phrase units. Accurate word segmentation is the basis for subsequent feature extraction and model training. After the server completes the word segmentation, it is also necessary to convert each word into a corresponding digital representation, namely, a word unit. Word unit conversion is the process of mapping words to unique word unit identifiers based on the vocabulary of the pre-trained model. Finally, the server converts each text sequence into a sequence of word unit identifiers, namely a word unit sequence, for example, token_ids = [token_id_1, token_id_2,…, token_id_n].

[0073] Step S202, extracting the embedding vector of the text sequence, the embedding vector includes a word embedding vector, a character embedding vector, a vocabulary embedding vector and a syntactic embedding vector.

[0074] Optionally, in the disclosed embodiments, the embedding vector in natural language processing refers to the representation of the basic units (such as words, characters, etc.) in the text sequence into low-dimensional vectors, which is intended to allow computers to better understand and process the semantic and grammatical information of the text. Among them, the embedding vector includes word embedding vector, character embedding vector, vocabulary embedding vector and syntactic embedding vector. The word embedding vector is used to represent the semantic information and contextual information of the word; the character embedding vector is used to represent the features at the character level, and can be used to process unregistered words (words not in the vocabulary) and morphologically complex words; the vocabulary embedding vector is used to represent the overall semantics, morphological features and subword information of the word; the syntactic embedding vector is used to represent the dependency relationship and grammatical structure information of the words in the sentence.

[0075] Specifically, the server uses a pre-trained word embedding model to extract word embedding vectors of a text sequence, such as using bert-finetuned-ner-word-embedding-model to map words to a low-dimensional vector space to obtain semantic representations of words and capture contextual information of the vocabulary; uses a pre-trained character embedding model to extract character embedding vectors of a text sequence, such as using roberta-classical-chinese-base-char to map characters to a low-dimensional vector space to obtain feature representations at the character level, and can be used to process unregistered words (vocabulary not in the vocabulary) and morphologically complex vocabulary; uses a pre-trained vocabulary embedding model to extract vocabulary embedding vectors of a text sequence, such as using the PhoBert_Lexical model to obtain the overall semantic representation of the vocabulary, capturing the morphological features and sub-word information of the vocabulary; uses a pre-trained dependency syntactic analyzer to extract syntactic embedding vectors of a text sequence, such as using synber to obtain the dependency relationship between words in a sentence, capturing the grammatical structure information of the sentence, and assisting the model in semantic understanding.

[0076] Step S203, fusing the word embedding vector, the character embedding vector and the vocabulary embedding vector to obtain a first fused vector, and extracting a global feature tensor and a local feature tensor of the first fused vector.

[0077] Optionally, in the embodiment of the present disclosure, if Figure 1 As shown, the server maps the word embedding vector, the character embedding vector, and the vocabulary embedding vector through a linear layer, and concatenates the mapped vectors to obtain a first fused vector. The formula of the first fused vector is as follows:

[0078] x i =[e wi ;e cj ;e lexi ]

[0079] Among them, x irepresents the first fusion vector, e wi represents the word embedding vector, e cj represents the character embedding vector, e lexi Represents the vocabulary embedding vector.

[0080] In addition, during the training phase of the model, the server adopts a multi-view contrastive learning method, using word embedding vectors, character embedding vectors, and vocabulary embedding vectors as different "views" to learn more discriminative feature representations by maximizing the consistency between representations of different views of the same sample and minimizing the consistency between representations of different samples.

[0081] Specifically, the server uses the multi-view contrast loss function to calculate the multi-view contrast loss and uses it as an auxiliary loss function to optimize the parameters of the model. The formula of the multi-view contrast loss function (calculate_multiview_contrastive_loss) is as follows:

[0082] L_contrastive=-log(exp(sim(z_w,z_c) / τ) / Σexp(sim(z_w,z_k) / τ))

[0083] -log(exp(sim(z_w,z_l) / τ) / Σexp(sim(z_w,z_k) / τ))

[0084] -log(exp(sim(z_c,z_l) / τ) / Σexp(sim(z_c,z_k) / τ))

[0085] Among them, sim represents the similarity measurement function, τ represents the temperature hyperparameter, z_w represents the vector after the word embedding vector is mapped, z_c represents the vector after the character embedding vector is mapped, z_l represents the vector after the vocabulary embedding vector is mapped, and z_k represents the view representation of other samples.

[0086] like Figure 1 As shown, the server inputs the first fusion vector into the global feature extraction module and the local feature extraction module respectively, and extracts the global feature tensor and the local feature tensor of the first fusion vector.

[0087] In some optional implementations, extracting the global feature tensor of the first fusion vector includes:

[0088] Based on the multi-head self-attention mechanism, the first fusion vector is converted into a first intermediate vector, and the first intermediate vector represents the dependency relationship between different positions in the text sequence;

[0089] Based on the first structured state space model, the first intermediate vector is converted into a second intermediate vector, where the second intermediate vector represents a global context representation of the text sequence;

[0090] Based on the feedforward neural network, the second intermediate vector is transformed to obtain the global feature tensor.

[0091] Optionally, in the embodiment of the present disclosure, if Figure 1 As shown, the global feature extraction module includes multiple multi-head self-attention sub-modules, residual connections and layer normalization, a first structured state space model, and a feedforward neural network.

[0092] The multi-head self-attention mechanism is an extension of the self-attention mechanism, which allows the model to perform self-attention calculations in different representation subspaces simultaneously. Each "head" learns to capture different features or patterns in the sequence, which can enhance the model's understanding and learning ability of the data. In the multi-head self-attention mechanism, the input sequence is first mapped into multiple different representation spaces (i.e., "heads"), and then the attention weights are calculated in these representation spaces separately. Each head generates a weighted output, which are then spliced ​​together to form a final output representation.

[0093] The structured state space model is a traditional mathematical framework that has been applied to deep learning in recent years. It can be used to describe the dynamic behavior of the system over time and can effectively capture complex dependencies in sequence data. The first structured state space model used in this embodiment is the Mamba model. The Mamba model includes the following steps: input gating (controlling the influence of the current input on the hidden state), candidate hidden states (calculating potential new hidden states), forgetting gating (controlling the influence of the previous hidden state on the current hidden state), hidden state update (updating the current hidden state based on input gating, candidate hidden states and forgetting gating), output gating (controlling the influence of the current hidden state on the output) and Mamba output (calculating the output of the Mamba model based on output gating and the current hidden state).

[0094] Specifically, the server first inputs the first fusion vector into the global feature extraction module, captures the dependency between any two positions in the text sequence through the multi-head self-attention submodule, extracts global, context-related features, and converts the first fusion vector into the first intermediate vector. The formula of the multi-head self-attention mechanism is as follows:

[0095]

[0096] Among them, Q represents the query vector, K represents the key vector, V represents the value vector, and d k Indicates the dimension of the key vector.

[0097] Then, the server inputs the first intermediate vector into the first structured state space model, further models the global context through the first structured state space model, and converts the first intermediate vector into the second intermediate vector. The calculation process of the first structured state space model is as follows:

[0098] h(t)=Ah(t)+Bx(t)h(t)=Ah(t)+Bx(t)

[0099] y(t)=Ch(t)+Dx(t)y(t)=Ch(t)+Dx(t)

[0100] Among them, h(t) represents the hidden state at time step t, x(t) represents the input, y(t) represents the output, and A, B, C, and D represent learnable matrices. Matrix A is used to describe the dependencies between the hidden states h(t) themselves, capturing and integrating long-distance dependency information in the sequence. Matrix B is used to map the current input x(t) to the hidden state space, combined with the existing hidden state h(t), and effectively integrate the input information into the hidden state. Matrix C is used to map the hidden state h(t) to the output space and generate task-related output representations. Matrix D is used to directly map the current input x(t) to the output space.

[0101] After the first structured state space model, the global feature extraction module also adds a feedforward neural network to further enhance the model's expressiveness. The second intermediate vector is converted into a global feature tensor through the feedforward neural network. The feedforward neural network usually consists of two linear layers, with an activation function used in the middle for nonlinear transformation. The formula of the feedforward neural network is as follows:

[0102] FFN(x)=W2*activation(W1*x+b1)+b2

[0103] Among them, x represents the output of the first structured state-space model, W1 and W2 represent the weight matrices of the two linear layers, b1 and b2 represent the bias terms of the two linear layers, and activation represents the activation function (such as ReLU, GELU, etc.).

[0104] In addition, the global feature extraction module also includes multiple residual connections and layer normalization submodules. The formula of the residual connection is as follows:

[0105] y = LayerNorm(x + SubLayer(x))

[0106] Among them, x represents input, SubLayer represents sub-layer operation, such as multi-head self-attention mechanism or the first structured state space model, and y represents output.

[0107] The formula for layer normalization is as follows:

[0108]

[0109] Among them, μ represents the mean, σ represents the standard deviation, and γ and β are learnable parameters.

[0110] In the disclosed embodiment, by converting the first fusion vector into the first intermediate vector based on the multi-head self-attention mechanism, the dependency between any two positions in the text sequence can be captured, thereby effectively capturing long-distance context information, so that the model can better understand the relationship between each word in the sentence. By converting the first intermediate vector into the second intermediate vector based on the first structured state space model, global context modeling can be efficiently performed, and it has good parallelism, thereby improving computational efficiency. By converting the second intermediate vector based on a feedforward neural network to obtain a global feature tensor, the nonlinear expression ability of the model can be enhanced, and the global context information of the text can be obtained.

[0111] In some optional implementations, extracting a local feature tensor of the first fused vector includes:

[0112] Based on a plurality of convolution kernels of different sizes, a convolution operation is performed on the first fusion vector to obtain a plurality of different feature maps;

[0113] Concatenate multiple different feature maps to obtain a local feature tensor.

[0114] Optionally, in the embodiment of the present disclosure, if Figure 1 As shown in Figure 3, the local feature extraction module includes a multi-layer convolutional neural network, a non-linear activation, a batch normalization layer, and a maximum pooling layer.

[0115] Specifically, the server uses multiple convolution kernels of different widths in parallel to perform convolution operations on the first fusion vector in the local feature extraction module to extract local features of different granularities (such as n-gram features) to obtain multiple different feature maps.

[0116] For example, the server performs convolution operations on the first fusion vector using three different sizes of convolution kernels, 3×3, 5×5, and 7×7, respectively, and obtains three different feature maps h_3, h_5, and h_7, where h_3 = Conv3×3(Xi, W_3), h_5 = Conv5×5(Xi, W_5), and h_7 = Conv7×7(Xi, W_7), W_3 represents the weight matrix of the 3×3 convolution kernel, W_5 represents the weight matrix of the 5×5 convolution kernel, and W_7 represents the weight matrix of the 7×7 convolution kernel, and the step size of each convolution kernel is 1.

[0117] In addition, the server performs a batch normalization operation after each convolution operation to accelerate training and improve the generalization ability of the model. Each branch after the batch normalization operation applies a maximum pooling operation to reduce the feature dimension and extract the most significant features. The outputs of multiple branches after the maximum pooling operation are then fused by element-wise addition, and a batch normalization operation is performed again to further improve model performance.

[0118] Specifically, the server performs batch normalization operations on the feature maps h_3, h_5, and h_7 obtained by each convolutional layer to obtain h_3_bn, h_5_bn, and h_7_bn. The normalization formula is as follows:

[0119]

[0120] Among them, μ B and σ B represents the mean and standard deviation of a mini-batch, and γ and β are learnable parameters.

[0121] Then, the server performs maximum pooling operations on the batch normalized feature maps h_3_bn, h_5_bn, and h_7_bn respectively. The size of the pooling window is 2×2, the step size is 2, and the pooling formula is as follows:

[0122] h_i_pool=MaxPool(h_i_bn)

[0123] Next, the server concatenates the features h_3_pool, h_5_pool, and h_7_pool after maximum pooling in the channel dimension to obtain the fused feature tensor h_local. The formula is as follows:

[0124] h_local=Concat(h_3_pool,h_5_pool,h_7_pool)

[0125] Finally, the server normalizes the fused feature tensor h_local to obtain the local feature tensor Local_Features.

[0126] In the disclosed embodiment, by performing a convolution operation on the first fusion vector to obtain a plurality of different feature maps, the local features of the text sequence can be effectively extracted, thereby capturing the local context information in the text sequence, which helps the model capture the local features inside the entity. By using convolution kernels of different sizes, local context information of different ranges can be captured, so that the model can better adapt to entities of different lengths and obtain the local context information of the text.

[0127] Step S204, feature enhancement is performed on the syntactic embedding vector to obtain a deep grammatical feature vector.

[0128] Optionally, in the embodiment of the present disclosure, if Figure 1 As shown, the server inputs the syntactic embedding vector into the deep grammatical feature extraction module to extract the deep grammatical features of the syntactic embedding vector.

[0129] In some optional implementations, the text sequence is a text sequence in the target domain, and the syntactic embedding vector is feature enhanced to obtain a deep grammatical feature vector, including:

[0130] Encoding the syntactic embedding vector through a private bidirectional long short-term memory network to obtain a first encoding vector, wherein the first encoding vector includes grammatical feature information of the target domain;

[0131] The syntactic embedding vector is encoded by a shared bidirectional long short-term memory network to obtain a second encoding vector, wherein the second encoding vector includes grammatical feature information shared by multiple domains;

[0132] Based on the second structured state space model, converting the first encoding vector and the second encoding vector to obtain a third intermediate vector;

[0133] The first encoding vector, the second encoding vector and the third intermediate vector are fused to obtain a deep grammatical feature vector.

[0134] Optionally, in the embodiment of the present disclosure, if Figure 1 As shown, the deep grammatical feature extraction module includes a private bidirectional long short-term memory network, a shared bidirectional long short-term memory network and a second structured state space model.

[0135] Both the private bidirectional long short-term memory network and the shared bidirectional long short-term memory network are composed of a forward long short-term memory network and a backward long short-term memory network. The forward long short-term memory network processes data forward according to the input sequence, and the backward long short-term memory network processes it in reverse. The two can capture the dependency relationship between each word in the sentence and the words before and after it, and understand the sentence semantics and grammatical structure more comprehensively. The private bidirectional long short-term memory network is configured separately for each field, and its parameters are optimized according to the training of specific field data and are only used in the field to which it belongs, which enables it to extract more targeted grammatical features for specific fields. The parameters of the shared bidirectional long short-term memory network are shared between all fields, and the same set of model parameters are used when training and applying on data in different fields. The second structured state space model used in this embodiment is the Mamba model.

[0136] Specifically, the server inputs the syntactic embedding vector into the private bidirectional long short-term memory network for encoding, extracts the grammatical features of the target domain, and obtains a first encoding vector whose dimension is consistent with the hidden layer size of the private bidirectional long short-term memory network. At the same time, the server inputs the syntactic embedding vector into the shared bidirectional long short-term memory network for encoding, extracts the grammatical features shared across domains, and obtains a second encoding vector whose dimension is consistent with the hidden layer size of the shared bidirectional long short-term memory network.

[0137] Then, the server inputs the first encoding vector and the second encoding vector into the second structured state space model for conversion, and uses the Mamba model's modeling ability for long-distance dependencies to further enhance the feature representation, and obtains a third intermediate vector whose dimension is consistent with the size of the hidden layer of the Mamba model. Finally, the server adds the first encoding vector, the second encoding vector, and the third intermediate vector element by element, fuses the grammatical features of the target domain, the grammatical features shared across domains, and the long-distance dependency information extracted by the Mamba model, and normalizes the added result through the normalization layer to obtain the final deep grammatical feature vector, whose dimension is consistent with the output dimension of the model configuration.

[0138] In the disclosed embodiment, the syntactic embedding vector is encoded by a private bidirectional long short-term memory network to obtain a first encoding vector, which can capture the dependency relationship between each word in the sentence and the words before and after it, and extract the grammatical features of the target domain. The syntactic embedding vector is encoded by a shared bidirectional long short-term memory network to obtain a second encoding vector, which can extract grammatical features shared across domains. By converting the first encoding vector and the second encoding vector based on the second structured state space model to obtain a third intermediate vector, the long-distance dependency relationship in the sequence can be efficiently captured. By fusing the first encoding vector, the second encoding vector and the third intermediate vector to obtain a deep grammatical feature vector, the syntactic information of the text can be obtained, the understanding of the grammatical structure of the text can be enhanced, and the accuracy of named entity recognition can be improved.

[0139] Step S205, generating a named entity recognition result of the text sequence based on the global feature tensor, the local feature tensor and the deep grammatical feature vector.

[0140] In some optional implementations, generating a named entity recognition result of a text sequence based on the global feature tensor, the local feature tensor, and the deep grammatical feature vector includes:

[0141] Concatenate the global feature tensor and the local feature tensor to obtain a first intermediate feature tensor;

[0142] Based on the attention weight matrix, the first intermediate feature tensor is weighted summed to obtain a fused feature tensor;

[0143] Concatenate the fused feature tensor and the deep syntax feature vector to obtain a second intermediate feature tensor;

[0144] Performing a nonlinear transformation on the second intermediate feature tensor to obtain a third intermediate feature tensor;

[0145] The third intermediate feature tensor is sequence labeled to obtain the named entity recognition result of the text sequence.

[0146] Optionally, in the embodiment of the present disclosure, if Figure 1 As shown, the server first concatenates the global feature tensor and the local feature tensor along the new dimension to obtain the first intermediate feature tensor, and reshapes the first intermediate feature tensor to meet the processing requirements of the subsequent cross-attention module.

[0147] Among them, the shapes of the global feature tensor and the local feature tensor are both (batch_size, sequence_length, hidden_size), which means there are batch_size samples, each sample contains sequence_length time steps, and the dimension of each feature is hidden_size. The concatenation operation is performed on the new dimension, and the shape of the generated first intermediate feature tensor is (batch_size, sequence_length, 2, hidden_size), which means there are two sub-feature tensors at each time step, namely the global feature tensor and the local feature tensor. The shape of the first intermediate feature tensor after reshaping is (batch_size*sequence_length, 2, hidden_size).

[0148] Then, if Figure 1 As shown, the server inputs the reshaped first intermediate feature tensor into the cross attention module, performs weighted summation on the first intermediate feature tensor based on the attention weight matrix, and obtains a fused feature tensor.

[0149] Specifically, the server performs weighted summation on the reshaped first intermediate feature tensor according to the attention weight matrix to obtain a fused feature tensor with a shape of (batch_size*sequence_length, hidden_size), and then restores the shape of the fused feature tensor to be consistent with the input data. Finally, the shape of the fused feature tensor is (batch_size, sequence_length, hidden_size).

[0150] In some optional embodiments, the first intermediate feature tensor includes sub-feature tensors of multiple time steps, and the attention weight matrix is ​​obtained based on the following method:

[0151] Based on the learned first weight matrix, feature mapping is performed on the first intermediate feature tensor to obtain a first mapped feature tensor;

[0152] Based on the learned second weight matrix, the first mapping feature tensor is linearly transformed to obtain the attention score of each sub-feature tensor;

[0153] The attention scores are averaged and normalized to obtain the attention weight matrix.

[0154] Optionally, in the embodiment of the present disclosure, the cross-attention mechanism adopts a weighted fusion mechanism based on linear transformation and Softmax function. The server projects the reshaped first intermediate feature tensor through a fully connected network including two linear layers and a hyperbolic tangent activation function based on the learned first weight matrix and the learned second weight matrix to obtain the attention scores of each sub-feature tensor, and uses the Softmax function to normalize the average calculated attention scores to obtain the attention weight matrix.

[0155] Specifically, the server inputs the reshaped first intermediate feature tensor into the cross attention module. For each time step t and element in batch b, a linear transformation layer and a hyperbolic tangent activation function are first used to map the input first intermediate feature tensor to the implicit space to obtain the first mapping feature tensor. The formula of the first mapping feature tensor is as follows:

[0156] a=Tanh(W_1*[h_t ∧ 1; h_t ∧ 2]+b_1

[0157] Among them, [h_t ∧ 1; h_t ∧ 2] represents the global feature h_t at the tth time step ∧ 1 and local feature h_t ∧ 2; W_1 represents the first learned weight matrix, with a shape of (2*hidden_size, hidden_size); b_1 is the learnable bias term, with a shape of (hidden_size); a represents the first mapping feature tensor, with a shape of (hidden_size).

[0158] Then, the server linearly transforms the first mapping feature tensor a again to obtain the attention score of each sub-feature tensor. The formula of the attention score is as follows:

[0159] e=W_2*a

[0160] Among them, W_2 represents the learned second weight matrix, with a shape of (hidden_size, 1) and no bias term, and e represents the attention score.

[0161] Next, the server averages the attention scores and normalizes them through the Softmax function to obtain the attention weight matrix. The normalization formula is as follows:

[0162] β=Softmax(Mean(e))

[0163] Among them, β represents the attention weight matrix.

[0164] In the embodiment of the present disclosure, by performing feature mapping and linear transformation on the first intermediate feature tensor based on the learned first weight matrix and second weight matrix, and performing averaging and normalization to obtain the attention weight matrix, it is possible to adaptively learn the important relationship between global features and local features, and dynamically weight them according to their importance, ultimately obtaining a more expressive fused feature tensor.

[0165] After obtaining the fused feature tensor, the server concatenates the fused feature tensor and the deep grammatical feature vector to obtain a second intermediate feature tensor, and then performs a nonlinear transformation on the second intermediate feature tensor to obtain a third intermediate feature tensor.

[0166] In some optional implementations, performing a nonlinear transformation on the second intermediate feature tensor to obtain a third intermediate feature tensor includes:

[0167] Based on the B-spline basis function, the second intermediate feature tensor is mapped into a second mapped feature tensor, and based on the learned third weight matrix, the second mapped feature tensor is linearly transformed to obtain a first enhanced feature tensor;

[0168] Performing linear transformation and nonlinear transformation on the second intermediate feature tensor in sequence to obtain a second enhanced feature tensor;

[0169] The first enhanced feature tensor and the second enhanced feature tensor are added to obtain a third intermediate feature tensor.

[0170] Optionally, in the disclosed embodiment, the server uses a learnable cubic spline function and linear transformation of a Kolmogorov-Arnold Network (KAN network) based on a B-spline basis function to perform nonlinear feature extraction. The B-spline basis function is determined by a predefined grid and spline order, and the input data is fitted by the least squares method to further enhance the feature representation. The KAN network is a neural network for nonlinear feature extraction, and each linear layer contains a basic linear transformation and a piecewise linear transformation based on B-splines, and the outputs of the two are added as the final output to achieve more refined enhanced features.

[0171] Specifically, the server first maps the second intermediate feature tensor to the B-spline basis function coefficient space, converts the representation space of the second intermediate feature tensor into the space of the B-spline basis function to obtain the B-spline basis function coefficients, and then applies the obtained B-spline basis function coefficients to the B-spline basis function to obtain the output of the B-spline basis function, i.e., the second mapped feature tensor.

[0172] Then, the server linearly transforms the second mapping feature tensor and the learned third weight matrix to obtain the output of the piecewise linear transformation, namely the first enhanced feature tensor, to further realize the nonlinear enhancement and dimensional mapping of the features. The formula of the first enhanced feature tensor is as follows:

[0173] output_spline=B(x)*W_spline

[0174] Among them, output_spline represents the first enhanced feature tensor, B(x) represents the second mapped feature tensor, and W_spline represents the learned third weight matrix.

[0175] Next, the server performs a linear transformation on the second intermediate feature tensor and activates it using a nonlinear activation function σ to obtain the output of the basic linear layer, i.e., the second enhanced feature tensor. The linear transformation and nonlinear activation are used to preliminarily enhance the feature representation and realize preliminary dimensional mapping. The formula of the second enhanced feature tensor is as follows:

[0176] output_base=σ(x*W_base+b_base)

[0177] Among them, output_base represents the second enhanced feature tensor, W_base represents a learnable parameter matrix, and b_base represents a learnable bias vector in the basis linear transformation.

[0178] Finally, the server adds the first enhanced feature tensor and the second enhanced feature tensor to obtain a third intermediate feature tensor with a shape of (batch_size, sequence_length, num_classes). The formula of the third intermediate feature tensor is as follows:

[0179] KAN_Features=output=output_base+output_spline

[0180] Among them, KAN_Features is the third intermediate feature tensor.

[0181] In the embodiment of the present disclosure, by mapping and linearly transforming the second intermediate feature tensor based on the B-spline basis function and the learned third weight matrix, and further performing linear transformation and nonlinear transformation, it is possible to more flexibly fit complex nonlinear relationships, thereby improving the feature expression ability of the model and better capturing complex semantic relationships in the text.

[0182] After obtaining the third intermediate feature tensor, the server performs sequence labeling on the third intermediate feature tensor to obtain a named entity recognition result of the text sequence.

[0183] In some optional implementations, performing sequence labeling on the third intermediate feature tensor to obtain a named entity recognition result of the text sequence includes:

[0184] The third intermediate feature tensor is input into a plurality of different conditional random field components respectively, and the conditional random field components respectively determine the score of each label sequence based on the third intermediate feature tensor and the learned label transfer probability and emission probability, and the label sequence represents the candidate named entity recognition result;

[0185] Based on the scores of each label sequence obtained by each conditional random field component, determine the target label sequence with the highest score;

[0186] Based on the target label sequence, the named entity recognition result of the text sequence is obtained.

[0187] Optionally, in an embodiment of the present disclosure, the server inputs the third intermediate feature tensor into a multi-head conditional random field layer, which includes multiple independent conditional random field heads. Each conditional random field head performs sequence labeling on the third intermediate feature tensor in parallel and learns the dependencies between different types of entity labels to obtain named entity recognition results for the text sequence.

[0188] Specifically, the server first uses the attention mechanism to weight the third intermediate feature tensor for each conditional random field head to obtain a weighted third intermediate feature tensor, thereby allowing the model to adaptively focus on features that are more relevant to the entity type corresponding to the current conditional random field head. The weighting process of the third intermediate feature tensor is as follows:

[0189] attended_features=KAN_Features*sigmoid(W_attn*KAN_Features)

[0190] Among them, attended_features represents the weighted third intermediate feature tensor, and W_attn represents the learnable weight matrix used to calculate the attention weight.

[0191] Then, the server calculates the score of each label sequence based on the weighted third intermediate feature tensor through each conditional random field head, using the learned label transfer probability and emission probability, and decodes the label sequence with the highest score through the Viterbi algorithm. The goal of each conditional random field head decoding is to optimize the following sequence score:

[0192] score(y)=∑ i Emission_Scores(x i ,y i )+∑ i Transition_Matrix(y i-1 ,y i )

[0193] Among them, x i Represents the weighted third intermediate feature tensor of the input, y i represents the predicted label, ∑ i Emission_Scores(x i ,y i ) represents the total emission score, which is used to comprehensively evaluate the overall matching degree between the input features and the predicted label sequence by accumulating the emission scores of all positions in the entire sample sequence; ∑ i Transition_Matrix(y i-1 ,y i ) represents the sum of transfer scores. The rationality of the label sequence is evaluated by summing the transfer scores between adjacent labels in the entire label sequence.

[0194] Finally, the server integrates the decoding results of multiple conditional random field headers to obtain the named entity recognition result.

[0195] In addition, during the training process, the server sums the losses of multiple conditional random field heads and uses them for model training to optimize model parameters.

[0196] In the disclosed embodiment, by using multiple different conditional random field components for sequence labeling, it is possible to learn the dependencies between different entity labels, and obtain better labeling results through global decoding, thereby improving the accuracy of named entity recognition.

[0197] In the disclosed embodiment, by performing weighted summation on the first intermediate feature tensor based on the attention weight matrix to obtain a fused feature tensor, global features and local features can be effectively fused, thereby better utilizing the contextual information of the text. By performing a nonlinear transformation on the second intermediate feature tensor to obtain a third intermediate feature tensor, features can be further enhanced, more complex semantic relationships can be captured, and more discriminative features can be provided for sequence labeling tasks.

[0198] In order to verify the effectiveness of the present disclosure in the field of biomedical entity recognition, the server selected different models for verification on two biomedical entity recognition datasets, NCBI-disease and BC2G, and obtained the corresponding F1 scores. The F1 score combines the precision and recall rate, and can comprehensively evaluate the accuracy of the model in identifying biomedical named entities. The F1 scores of different models on the NCBI-disease and BC2GM datasets are shown in Table 1:

[0199] Table 1 F1 scores of different models on NCBI-disease and BC2GM datasets

[0200]

[0201]

[0202] As can be seen from Table 1, the F1 score of the model disclosed in the present invention on the NCBI-disease dataset is 93.27, and the F1 score on the BC2GM dataset is 94.43, both of which are higher than other models, and the precision and accuracy of named entity recognition are the best.

[0203] In an embodiment of the present disclosure, a text sequence to be recognized is obtained; an embedding vector of the text sequence is extracted, and the embedding vector includes a word embedding vector, a character embedding vector, a vocabulary embedding vector, and a syntax embedding vector; the word embedding vector, the character embedding vector, and the vocabulary embedding vector are fused to obtain a first fused vector, and a global feature tensor and a local feature tensor of the first fused vector are extracted; the syntax embedding vector is feature enhanced to obtain a deep grammatical feature vector; based on the global feature tensor, the local feature tensor, and the deep grammatical feature vector, a named entity recognition result of the text sequence is generated. Since the embodiment of the present disclosure fuses the global feature tensor and the local feature tensor, the contextual information of the text can be effectively utilized, and the deep grammatical feature vector is added to enhance the understanding of the grammatical structure, thereby improving the recognition accuracy of the named entity.

[0204] In this embodiment, a named entity recognition device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0205] This embodiment provides a named entity recognition device, such as Figure 3 As shown, including:

[0206] An acquisition module 301 is used to acquire a text sequence to be recognized;

[0207] A first extraction module 302, used to extract an embedding vector of a text sequence, wherein the embedding vector includes a word embedding vector, a character embedding vector, a vocabulary embedding vector and a syntactic embedding vector;

[0208] A second extraction module 303 is used to fuse the word embedding vector, the character embedding vector and the vocabulary embedding vector to obtain a first fused vector, and extract a global feature tensor and a local feature tensor of the first fused vector;

[0209] An enhancement module 304 is used to perform feature enhancement on the syntactic embedding vector to obtain a deep grammatical feature vector;

[0210] The generation module 305 is used to generate a named entity recognition result of the text sequence based on the global feature tensor, the local feature tensor and the deep grammatical feature vector.

[0211] In an embodiment of the present disclosure, a text sequence to be recognized is obtained; an embedding vector of the text sequence is extracted, and the embedding vector includes a word embedding vector, a character embedding vector, a vocabulary embedding vector, and a syntax embedding vector; the word embedding vector, the character embedding vector, and the vocabulary embedding vector are fused to obtain a first fused vector, and a global feature tensor and a local feature tensor of the first fused vector are extracted; the syntax embedding vector is feature enhanced to obtain a deep grammatical feature vector; based on the global feature tensor, the local feature tensor, and the deep grammatical feature vector, a named entity recognition result of the text sequence is generated. Since the embodiment of the present disclosure fuses the global feature tensor and the local feature tensor, the contextual information of the text can be effectively utilized, and the deep grammatical feature vector is added to enhance the understanding of the grammatical structure, thereby improving the recognition accuracy of the named entity.

[0212] In some optional implementations, the second extraction module 303 includes:

[0213] A first conversion submodule, used for converting the first fusion vector into a first intermediate vector based on a multi-head self-attention mechanism, where the first intermediate vector represents the dependency relationship between different positions in the text sequence;

[0214] A second conversion submodule, configured to convert the first intermediate vector into a second intermediate vector based on the first structured state space model, wherein the second intermediate vector represents a global context representation of the text sequence;

[0215] The third conversion submodule is used to convert the second intermediate vector based on a feedforward neural network to obtain a global feature tensor.

[0216] In some optional implementations, the second extraction module 303 includes:

[0217] A convolution submodule, configured to perform a convolution operation on the first fused vector based on a plurality of convolution kernels of different sizes to obtain a plurality of different feature maps;

[0218] The first splicing submodule is used to splice multiple different feature maps to obtain a local feature tensor.

[0219] In some optional implementations, the enhancement module 304 includes:

[0220] A first encoding submodule is used to encode the syntactic embedding vector through a private bidirectional long short-term memory network to obtain a first encoding vector, wherein the first encoding vector includes grammatical feature information of the target domain;

[0221] A second encoding submodule is used to encode the syntactic embedding vector through a shared bidirectional long short-term memory network to obtain a second encoding vector, wherein the second encoding vector includes grammatical feature information shared by multiple fields;

[0222] A fourth conversion submodule, configured to convert the first encoding vector and the second encoding vector based on the second structured state space model to obtain a third intermediate vector;

[0223] The fusion submodule is used to fuse the first encoding vector, the second encoding vector and the third intermediate vector to obtain a deep grammatical feature vector.

[0224] In some optional implementations, the generation module 305 includes:

[0225] A second splicing submodule is used to splice the global feature tensor and the local feature tensor to obtain a first intermediate feature tensor;

[0226] A weighted submodule, used for performing weighted summation on the first intermediate feature tensor based on the attention weight matrix to obtain a fused feature tensor;

[0227] A third concatenation submodule is used to concatenate the fused feature tensor and the deep syntax feature vector to obtain a second intermediate feature tensor;

[0228] A transformation submodule, used for performing a nonlinear transformation on the second intermediate feature tensor to obtain a third intermediate feature tensor;

[0229] The labeling submodule is used to perform sequence labeling on the third intermediate feature tensor to obtain the named entity recognition result of the text sequence.

[0230] In some optional implementations, the weighting submodule includes:

[0231] A mapping unit, configured to perform feature mapping on the first intermediate feature tensor based on the learned first weight matrix to obtain a first mapped feature tensor;

[0232] A first transformation unit, configured to perform a linear transformation on the first mapping feature tensor based on the learned second weight matrix to obtain an attention score of each sub-feature tensor;

[0233] The averaging unit is used to average and normalize the attention scores to obtain the attention weight matrix.

[0234] In some optional implementations, the transformation submodule includes:

[0235] A second transformation unit is used to map the second intermediate feature tensor into a second mapping feature tensor based on a B-spline basis function, and to perform a linear transformation on the second mapping feature tensor based on a learned third weight matrix to obtain a first enhanced feature tensor;

[0236] A third transformation unit is used to perform linear transformation and nonlinear transformation on the second intermediate feature tensor in sequence to obtain a second enhanced feature tensor;

[0237] The adding unit is used to add the first enhanced feature tensor and the second enhanced feature tensor to obtain a third intermediate feature tensor.

[0238] In some optional implementations, the annotation submodule includes:

[0239] A first determination unit is used to input the third intermediate feature tensor into a plurality of different conditional random field components respectively, and the conditional random field components respectively determine the scores of each label sequence based on the third intermediate feature tensor and the learned label transfer probability and emission probability, and the label sequence represents the candidate named entity recognition result;

[0240] A second determination unit is used to determine a target label sequence with the highest score based on the scores of each label sequence obtained by each conditional random field component;

[0241] The obtaining unit is used to obtain the named entity recognition result of the text sequence based on the target label sequence.

[0242] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0243] The named entity recognition device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0244] The present disclosure also provides a computer device having the above Figure 3 The named entity recognition device shown.

[0245] See also Figure 4 , Figure 4 is a schematic diagram of a computer device provided by an optional embodiment of the present disclosure, such as Figure 4As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 4 A processor 10 is taken as an example.

[0246] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0247] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0248] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0249] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0250] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0251] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium and downloaded through a network, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0252] A part of the present disclosure may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present disclosure through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer-readable media includes, but is not limited to, source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer.

[0253] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for named entity recognition, characterized in that: The method comprises: Get the text sequence to be recognized; Extracting embedding vectors of the text sequence, wherein the embedding vectors include word embedding vectors, character embedding vectors, vocabulary embedding vectors, and syntactic embedding vectors; Fusing the word embedding vector, the character embedding vector and the vocabulary embedding vector to obtain a first fused vector, and extracting a global feature tensor and a local feature tensor of the first fused vector; Performing feature enhancement on the syntactic embedding vector to obtain a deep grammatical feature vector; Based on the global feature tensor, the local feature tensor and the deep grammatical feature vector, a named entity recognition result of the text sequence is generated.

2. The method according to claim 1, characterized in that The extracting the global feature tensor of the first fusion vector includes: Based on a multi-head self-attention mechanism, converting the first fusion vector into a first intermediate vector, where the first intermediate vector represents the dependency relationship between different positions in the text sequence; Based on a first structured state space model, converting the first intermediate vector into a second intermediate vector, wherein the second intermediate vector represents a global context representation of the text sequence; Based on a feedforward neural network, the second intermediate vector is transformed to obtain the global feature tensor.

3. The method according to claim 1, characterized in that The extracting the local feature tensor of the first fusion vector includes: Based on a plurality of convolution kernels of different sizes, a convolution operation is performed on the first fusion vector to obtain a plurality of different feature maps; The multiple different feature maps are concatenated to obtain the local feature tensor.

4. The method according to claim 1, characterized in that: The text sequence is a text sequence in the target domain, and the feature enhancement of the syntactic embedding vector to obtain a deep grammatical feature vector includes: Encoding the syntactic embedding vector through a private bidirectional long short-term memory network to obtain a first encoding vector, wherein the first encoding vector includes grammatical feature information of the target domain; Encoding the syntactic embedding vector through a shared bidirectional long short-term memory network to obtain a second encoding vector, wherein the second encoding vector includes grammatical feature information shared by multiple domains; Based on a second structured state space model, transforming the first encoding vector and the second encoding vector to obtain a third intermediate vector; The first encoding vector, the second encoding vector and the third intermediate vector are fused to obtain the deep grammatical feature vector.

5. The method according to claim 1, characterized in that The generating a named entity recognition result of the text sequence based on the global feature tensor, the local feature tensor and the deep grammatical feature vector comprises: Concatenating the global feature tensor and the local feature tensor to obtain a first intermediate feature tensor; Based on the attention weight matrix, performing weighted summation on the first intermediate feature tensors to obtain a fused feature tensor; Concatenating the fused feature tensor and the deep syntax feature vector to obtain a second intermediate feature tensor; Performing a nonlinear transformation on the second intermediate feature tensor to obtain a third intermediate feature tensor; The third intermediate feature tensor is sequence labeled to obtain a named entity recognition result of the text sequence.

6. The method according to claim 5, characterized in that The first intermediate feature tensor includes sub-feature tensors of multiple time steps, and the attention weight matrix is ​​obtained based on the following method: Based on the learned first weight matrix, feature mapping is performed on the first intermediate feature tensor to obtain a first mapped feature tensor; Based on the learned second weight matrix, linearly transform the first mapping feature tensor to obtain an attention score of each of the sub-feature tensors; The attention scores are averaged and normalized to obtain the attention weight matrix.

7. The method according to claim 5, characterized in that The performing a nonlinear transformation on the second intermediate feature tensor to obtain a third intermediate feature tensor includes: Based on a B-spline basis function, mapping the second intermediate feature tensor into a second mapping feature tensor, and based on the learned third weight matrix, performing a linear transformation on the second mapping feature tensor to obtain a first enhanced feature tensor; performing linear transformation and nonlinear transformation on the second intermediate feature tensor in sequence to obtain a second enhanced feature tensor; The first enhanced feature tensor and the second enhanced feature tensor are added to obtain the third intermediate feature tensor.

8. The method according to claim 5, characterized in that The step of performing sequence labeling on the third intermediate feature tensor to obtain a named entity recognition result of the text sequence includes: Inputting the third intermediate feature tensor into a plurality of different conditional random field components respectively, and determining scores of respective label sequences based on the third intermediate feature tensor and the learned label transfer probability and emission probability respectively by the conditional random field components, wherein the label sequences represent candidate named entity recognition results; Determine a target tag sequence with the highest score based on the scores of each of the tag sequences obtained by each of the conditional random field components; Based on the target tag sequence, a named entity recognition result of the text sequence is obtained.

9. A named entity recognition device, characterized in that: The device comprises: An acquisition module, used for acquiring a text sequence to be recognized; A first extraction module, configured to extract an embedding vector of the text sequence, wherein the embedding vector includes a word embedding vector, a character embedding vector, a vocabulary embedding vector, and a syntactic embedding vector; A second extraction module, configured to fuse the word embedding vector, the character embedding vector and the vocabulary embedding vector to obtain a first fused vector, and extract a global feature tensor and a local feature tensor of the first fused vector; An enhancement module, used for performing feature enhancement on the syntactic embedding vector to obtain a deep grammatical feature vector; A generation module is used to generate a named entity recognition result of the text sequence based on the global feature tensor, the local feature tensor and the deep grammatical feature vector.

10. A computer program product, characterized in that The method comprises computer instructions, wherein the computer instructions are used to cause a computer to execute the method for named entity recognition according to any one of claims 1 to 8.

Images