Named entity recognition method and device and electronic equipment
Through adversarial learning and adaptive gradient optimization techniques, the generalization ability of named entity recognition models is improved, and the problem of limited generalization ability of models in the existing technology is solved.
Patent Information
- Application Number
- CN202411089057.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2024-11-08
AI Technical Summary
The training mechanism of existing named entity recognition models is incomplete, resulting in limited generalization ability when identifying untrained data or data slightly deviating from the training distribution.
Adversarial learning method is used to train named entity recognition models, process embedded vector sequences through two-way long and short-term memory networks, discard layers and conditional random fields layers, generate sparsely activated feature vectors and feature integration, and use adversarial samples and adaptive gradient clipping. Optimize model parameters with gradient centralized components.
It significantly improves the generalization ability of the model, can more accurately identify untrained text data, and solves the problem of limited generalization ability.
Smart Images

Figure CN118917314A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of natural language processing, and in particular to a method, device and electronic device for named entity recognition. Background Art
[0002] With the rapid development of artificial intelligence technology, especially the update and iteration of natural language processing, named entity recognition has become an important technology in natural language processing. At present, a hybrid learning Bi-LSTM-CRF method has been proposed in the relevant technology. This method mixes the statistical conditional random field (CRF) with the bidirectional long short-term memory network (Bi-LSTM) based on deep learning. However, this hybrid learning method does not make further improvements to the training mechanism of the model, resulting in limited generalization ability of the model.
[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention
[0004] The embodiments of the present application provide a method, apparatus, and electronic device for named entity recognition to at least solve the technical problem that the training mechanism of the named entity recognition model in the related art is not perfect, resulting in limited generalization ability of the model when recognizing untrained data or data that slightly deviates from the training distribution.
[0005] According to one aspect of an embodiment of the present application, a method for named entity recognition is provided, including: obtaining a text sequence to be recognized; using an embedded text vector layer in an entity recognition model to vectorize the text sequence to be recognized to obtain an embedded vector sequence; using a hidden layer in the entity recognition model to recognize named entities in the embedded vector sequence to obtain named entities in the text sequence to be recognized, wherein the entity recognition model is a model trained using an adversarial learning method.
[0006] In some embodiments of the present application, a hidden layer in an entity recognition model is used to recognize named entities in an embedded vector sequence to obtain named entities of a text sequence to be recognized, including: using a bidirectional long short-term memory network layer in the entity recognition model to process the embedded vector sequence to obtain a feature vector sequence containing a forward hidden state and a reverse hidden state; using a discard layer of the entity recognition model to randomly discard some elements in the feature vector sequence according to a preset probability to obtain a sparsely activated feature vector sequence; using a fully connected layer in the entity recognition model to perform feature integration on feature vectors in the sparsely activated feature vector sequence to obtain a prediction vector sequence of fixed dimension; using a conditional random field layer of the entity recognition model to perform label prediction on the named entities in the prediction vector sequence to obtain named entities of the text sequence to be recognized.
[0007] In some embodiments of the present application, the entity recognition model is trained in the following manner: obtaining a sample sequence, wherein the sample sequence includes a text sequence and a real annotation sequence corresponding to the named entities in the text sequence; using the embedded text vector layer of the initial entity recognition model to vectorize the text sequence to obtain an embedded vector sequence of the sample sequence; using the bidirectional long short-term memory network layer in the entity recognition model to process the embedded vector sequence to obtain a feature vector sequence containing a forward hidden state and a reverse hidden state; using the discard layer of the entity recognition model to randomly discard some elements in the feature vector sequence according to a preset probability to obtain a sparsely activated feature vector sequence; using the fully connected layer in the entity recognition model to integrate the features of the feature vectors in the sparsely activated feature vector sequence to obtain a fixed-dimensional prediction vector sequence; using the conditional random field layer of the entity recognition model to perform annotation prediction on the named entities in the prediction vector sequence to obtain a prediction sample sequence; determining the loss function based on the predicted sample sequence and the real annotation sequence, and adjusting the model parameters of the initial entity recognition model based on the loss function.
[0008] In some embodiments of the present application, the model parameters of the initial entity recognition model are adjusted according to the loss function, including: determining the gradient of the model parameters of the entity recognition model according to the loss function; processing the sample sequence using an adversarial learning algorithm to obtain an adversarial sample; determining a first adversarial gradient based on the gradient generated by the adversarial sample and the gradient of the model parameters of the initial entity recognition model; using adaptive gradient clipping and gradient centering components to clip and center the first adversarial gradient to obtain a second adversarial gradient; updating the model parameters of the initial entity recognition model according to the second adversarial gradient to obtain an entity recognition model.
[0009] In some embodiments of the present application, after the first adversarial gradient is clipped and centralized using the adaptive gradient clipping and gradient centralization components, the method further includes: updating the current momentum vector according to the obtained parity of the current iteration round; when the current iteration round is an odd number, updating the current momentum vector according to the moment estimated average of the current momentum vector, and assigning a positive weight to the current momentum vector; when the current iteration round is an even number, updating the current momentum vector according to the moment estimated average of the previous momentum vector, and assigning a negative weight to the current momentum vector; and adjusting the model parameters of the initial entity recognition model according to the updated current momentum vector.
[0010] In some embodiments of the present application, updating the model parameters of the initial entity recognition model according to the second adversarial gradient includes: adjusting the parameter weights of the model parameters of the initial entity recognition model using an optimization strategy, wherein the optimization strategy includes at least one of: performing L2 regularization on the parameter weights using the L2 norm, clipping the parameter weights according to a preset ratio, and resetting the exponential moving average of the parameters to the current parameter value according to a preset step size; and updating the model parameters of the initial entity recognition model according to the adjusted parameter weights.
[0011] In some embodiments of the present application, after obtaining the sample sequence, the method further includes: truncating or padding a text sequence in the sample sequence according to a preset maximum sequence length; truncating or padding a tag sequence in the sample sequence according to the maximum sequence length, wherein the length of the text sequence after truncation or padding is consistent with the length of the tag sequence after truncation or padding.
[0012] According to another aspect of an embodiment of the present application, a training method for an entity recognition model is also provided, including: obtaining a sample sequence, wherein the sample sequence includes a text sequence and a real annotation sequence corresponding to a named entity in the text sequence; using an embedded text vector layer of an initial entity recognition model to vectorize the text sequence to obtain an embedded vector sequence of the sample sequence; using a bidirectional long short-term memory network layer in the entity recognition model to process the embedded vector sequence to obtain a feature vector sequence containing a forward hidden state and a reverse hidden state; using a discard layer of the entity recognition model to randomly discard some elements in the feature vector sequence according to a preset probability to obtain a sparsely activated feature vector sequence; using a fully connected layer in the entity recognition model to perform feature integration on feature vectors in the sparsely activated feature vector sequence to obtain a prediction vector sequence of fixed dimension; using a conditional random field layer of the entity recognition model to perform annotation prediction on the named entities in the prediction vector sequence to obtain a prediction sample sequence; determining a loss function based on the predicted sample sequence and the real annotation sequence, and adjusting the model parameters of the initial entity recognition model based on the loss function.
[0013] In some embodiments of the present application, the model parameters of the initial entity recognition model are adjusted according to the loss function, including: determining the gradient of the model parameters of the entity recognition model according to the loss function; processing the sample sequence using an adversarial learning algorithm to obtain an adversarial sample; determining a first adversarial gradient based on the gradient generated by the adversarial sample and the gradient of the model parameters of the initial entity recognition model; using adaptive gradient clipping and gradient centering components to clip and center the first adversarial gradient to obtain a second adversarial gradient; updating the model parameters of the initial entity recognition model according to the second adversarial gradient to obtain an entity recognition model.
[0014] According to another aspect of an embodiment of the present application, a device for named entity recognition is also provided, including: an acquisition module for acquiring a text sequence to be recognized; a conversion module for using an embedded text vector layer in an entity recognition model to vectorize the text sequence to be recognized to obtain an embedded vector sequence; a recognition module for using a hidden layer in the entity recognition model to recognize named entities in the embedded vector sequence to obtain named entities in the text sequence to be recognized, wherein the entity recognition model is an entity recognition model trained using an adversarial learning method.
[0015] According to another aspect of the embodiments of the present application, an electronic device is also provided, including: a memory for storing program instructions; a processor, connected to the memory, for executing program instructions to implement the following functions: obtaining a text sequence to be recognized; using an embedded text vector layer in an entity recognition model to vectorize the text sequence to be recognized to obtain an embedded vector sequence; using a hidden layer in the entity recognition model to recognize named entities in the embedded vector sequence to obtain named entities of the text sequence to be recognized, wherein the entity recognition model is an entity recognition model trained using an adversarial learning method.
[0016] According to another aspect of the embodiments of the present application, a non-volatile storage medium is provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned data platform monitoring method by running the computer program.
[0017] According to another aspect of the embodiments of the present application, a computer program product is provided, including computer instructions, which implement the above-mentioned data platform monitoring method when executed by a processor.
[0018] In an embodiment of the present application, the system obtains a text sequence to be recognized, uses an embedded text vector layer to convert the text sequence into a computer-understandable vector representation, and then uses an adversarially trained entity recognition model to perform named entity recognition, and finally obtains a recognition result. Since the generalization ability of the entity recognition model is significantly improved after adversarial training, it can more accurately recognize untrained text data, thereby solving the technical problem that the training mechanism of the named entity recognition model in the related technology is not perfect, resulting in limited generalization ability of the model when recognizing untrained data or data that slightly deviates from the training distribution. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0020] Figure 1is a hardware structure block diagram of a computer terminal for implementing a method for named entity recognition according to an embodiment of the present application;
[0021] Figure 2 is a flowchart of a method for implementing named entity recognition according to an embodiment of the present application;
[0022] Figure 3 is a flow chart of a method for implementing a training model for an entity recognition model according to an embodiment of the present application;
[0023] Figure 4 It is a schematic diagram of the overall process of a training method for an entity recognition model according to an embodiment of the present application;
[0024] Figure 5 is a structural schematic diagram of a device for named entity recognition according to an embodiment of the present application;
[0025] Figure 6 It is a structural diagram of a device for training an entity recognition model according to an embodiment of the present application. DETAILED DESCRIPTION
[0026] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0028] The information collected in the embodiments of the present application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or reject automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered.
[0029] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:
[0030] Natural Language Processing (NLP) is an important direction in the fields of computer science and artificial intelligence, which studies various theories and methods that can enable effective communication between humans and computers using natural language.
[0031] Named Entity Recognition (NER) is a subtask of information extraction in natural language processing, which aims to identify entities of interest in text, such as location, organization, and time. Identified entities can be used in various downstream applications, such as de-identification and information extraction systems based on patient records, and can also be used as features of machine learning systems for other natural language processing tasks. It is an important research area in natural language processing.
[0032] Bi-directional Long Short-Term Memory (Bi-LSTM): It is composed of a forward long short-term memory network (LSTM, a type of RNN (Recurrent Neural Network)) and a backward LSTM. Due to its design characteristics, it is very suitable for modeling time series data and is suitable for sequence labeling tasks with upper and lower relationships. Therefore, it is often used to model contextual information in NLP.
[0033] Conditional Random Field (CRF): It combines the characteristics of the maximum entropy model and the hidden Markov model. It is an undirected graph model suitable for discriminant models of prediction tasks, in which adjacent contextual information or states will affect the current prediction. It is often used to label or analyze sequence data, such as word segmentation, part-of-speech tagging, and named entity recognition in natural language processing sequence labeling tasks.
[0034] With the rapid development of artificial intelligence technology, named entity recognition has become an important technology in natural language processing. Its main purpose is to identify entity types with specific meanings from unstructured texts, including names of people, places, disease names, various proper nouns, etc. It is a prerequisite task for natural language processing technologies such as relationship extraction and question-answering systems. However, the rule-based named entity recognition method in the related art often depends on the specific language field and text style, and it is difficult to cover all language phenomena; the named entity recognition method based on conditional random field (CRF) has limited recognition ability because it does not deeply study the front and back position relationship of the text sequence and has not undergone deep learning of neural networks; the named entity recognition method based on bidirectional long short-term memory network (Bi-LSTM) performs bidirectional processing, and each output node contains both forward and reverse information, which is easy to form overfitting when used alone; and the Bi-LSTM-CRF named entity recognition method based on hybrid learning does not propose further improvements to the training mechanism, optimizer, and underlying representation model for enhancing the generalization ability of the model, resulting in limited model generalization ability and difficulty in scene use. In order to solve the above technical problems, the embodiments of the present application provide corresponding solutions, which are described in detail below.
[0035] The method embodiment of named entity recognition provided in the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a method for named entity recognition. Figure 1 As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions connected via a wired and / or wireless network. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art can understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.
[0036] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).
[0037] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method of named entity recognition in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned method of named entity recognition. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0038] The transmission module 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0039] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .
[0040] It should be noted that, in some optional embodiments, the above Figure 1 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.
[0041] Figure 2 is a flowchart of a method for named entity recognition according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:
[0042] Step S202, obtaining a text sequence to be recognized.
[0043] In the above step S202, the text sequence is a text sequence required for named entity recognition in any scenario, for example, an information text sequence crawled from a website, a text sequence manually input, an authoritative public data set, etc.
[0044] Step S204, using the embedded text vector layer in the entity recognition model to perform vectorization processing on the text sequence to be recognized, to obtain an embedded vector sequence.
[0045] In the above step S204, the embedded text vector layer is the input layer of the entity recognition model. In an optional embodiment, when recognizing Chinese named entities, the embedded text vector layer can be a Chinese pre-trained language model chinese-roberta-wwm-ext based on the Robustly Optimized BERT Pretraining Approach (RoBERTa for short) model.
[0046] In some embodiments, the embedded text vector layer can convert a text sequence into an embedded vector sequence through word embedding, character embedding, positional encoding, etc.
[0047] Step S206, using the hidden layer in the entity recognition model to recognize the named entities in the embedded vector sequence to obtain the named entities of the text sequence to be recognized, wherein the entity recognition model is a model trained by adversarial learning.
[0048] In the above step S206, the model for entity recognition is trained by adversarial learning, for example, adding perturbations to the embedding vectors of the input layer to generate adversarial samples for training, screening out difficult samples to generate adversarial samples for local adversarial training, etc.
[0049] Through the above steps S202 to S206, the system obtains the text sequence to be recognized, uses the embedded text vector layer to convert the text sequence into a vector representation that can be understood by the computer, and then uses the adversarially trained entity recognition model to perform named entity recognition, and finally obtains the recognition result. Since the generalization ability of the entity recognition model is significantly improved after adversarial training, it can more accurately recognize untrained text data, thereby solving the technical problem that the training mechanism of the named entity recognition model in the related technology is not perfect, resulting in the limited generalization ability of the model when recognizing untrained data or data that slightly deviates from the training distribution.
[0050] In some optional embodiments, the hidden layer includes a bidirectional long short-term memory network layer, a dropout layer, and a fully connected layer, and the named entity recognition result of the text sequence is output by a conditional random field layer.
[0051] Based on the structure of the above hidden layer, step S206 can be expressed as the following process, but is not limited to this: the bidirectional long short-term memory network layer in the entity recognition model is used to process the embedded vector sequence to obtain a feature vector sequence containing a forward hidden state and a reverse hidden state; the discard layer of the entity recognition model is used to randomly discard some elements in the feature vector sequence according to a preset probability to obtain a sparsely activated feature vector sequence; the fully connected layer in the entity recognition model is used to integrate the features of the feature vectors in the sparsely activated feature vector sequence to obtain a prediction vector sequence of fixed dimension; the conditional random field layer of the entity recognition model is used to label and predict the named entities in the prediction vector sequence to obtain the named entities of the text sequence to be recognized.
[0052] In an optional embodiment, a bidirectional long short-term memory network layer (Bidirectional Long Short-Term Memory, referred to as Bi-LSTM) includes two LSTM layers, namely a forward LSTM and a reverse LSTM, which are used to generate a forward hidden state and a reverse hidden state, respectively. Before outputting a forward hidden state sequence and a reverse hidden state sequence obtained according to an embedded vector sequence, the Bi-LSTM model performs feature fusion on the two hidden state sequences, for example, concatenating the forward and reverse hidden states at each time step, performing weighted averaging on the forward and reverse hidden states, or selecting the maximum value (i.e., maximum pooling) of the forward and reverse hidden states at each time step, etc., and uses the sequence after feature fusion as the final output feature vector sequence.
[0053] In the process of named entity recognition, the dropout layer introduces randomness, which may cause differences in the inference results for the same input each time. In order to obtain deterministic output, the dropout layer can be turned off, a lower dropout rate can be set, or other regularization techniques can be used instead to ensure the stability of the model.
[0054] The fully connected layer converts the high-dimensional feature vector from the bidirectional long short-term memory network layer or the discard layer into a low-dimensional representation suitable for the final task, that is, mapping the feature vector to a space with the same dimension as the number of entity categories, and obtaining a prediction vector with the same dimension as the number of entity categories. The number of entity categories can be determined by a predefined label set, and the prediction vector contains the probability or score of the entity corresponding to each predefined entity category.
[0055] Since the prediction vector captures the local features of the input text but does not fully consider the dependencies between entity labels corresponding to entity categories, a conditional random field (CRF) layer is used to introduce dependencies between labels. The received prediction vector sequence and the transfer matrix containing the probability of transition between different labels are decoded. For example, the Viterbi algorithm in the dynamic programming algorithm is used to find the most likely entity label sequence, and finally each entity in the text sequence and the corresponding entity category label are output.
[0056] In some embodiments of the present application, the entity recognition model is trained in the following manner: obtaining a sample sequence, wherein the sample sequence includes a text sequence and a real annotation sequence corresponding to the named entities in the text sequence; using the embedded text vector layer of the initial entity recognition model to vectorize the text sequence to obtain an embedded vector sequence of the sample sequence; using the bidirectional long short-term memory network layer in the entity recognition model to process the embedded vector sequence to obtain a feature vector sequence containing a forward hidden state and a reverse hidden state; using the discard layer of the entity recognition model to randomly discard some elements in the feature vector sequence according to a preset probability to obtain a sparsely activated feature vector sequence; using the fully connected layer in the entity recognition model to integrate the features of the feature vectors in the sparsely activated feature vector sequence to obtain a fixed-dimensional prediction vector sequence; using the conditional random field layer of the entity recognition model to perform annotation prediction on the named entities in the prediction vector sequence to obtain a prediction sample sequence; determining the loss function based on the predicted sample sequence and the real annotation sequence, and adjusting the model parameters of the initial entity recognition model based on the loss function.
[0057] In an optional embodiment, the sample sequence used for entity recognition model training is a sample sequence that has been preprocessed to facilitate model processing, for example, defining the entity vocabulary VOCAB, creating a mapping dictionary corresponding to annotation labels and integer indices, boundary annotation, data cleaning, etc.
[0058] The initial entity recognition model is an untrained entity recognition model, and the relevant parameters of the model are pre-set parameters or default parameter values. The relevant parameters are, for example, hidden layer dimension, embedded text vector dimension, Dropout probability, fully connected layer dimension, etc.
[0059] In some embodiments of the present application, the pre-trained model "chinese-roberta-wwm-ext" based on the Transformer architecture is used as the embedded text vector layer for deep language representation learning at the bottom layer to realize the encoding of the Chinese input text sequence. For example, the model can process the input text sequence again, convert it into a token sequence that the model can understand, add special tags at the beginning and end of the sequence, and then convert the token sequence into a numerical input that the model can understand, and finally obtain the embedded vector representation sequence of the text sequence.
[0060] The Bi-LSTM layer is used to capture the long-term dependencies (i.e., long-distance semantic information) in the text sequence, so as to obtain a more accurate encoding of the input text sequence based on the embedded text vector layer. For example, after obtaining the input embedding vector sequence, the forward LSTM processes the input sequence from left to right to capture the temporal dependencies in the sequence, and the reverse LSTM processes the input sequence from right to left to capture the reverse temporal dependencies. The hidden states of the forward and reverse LSTMs are then merged to obtain the bidirectional context information for each time step. The merged hidden states are converted into the final feature representation, i.e., the feature vector sequence, through the weight matrix of the Bi-LSTM layer.
[0061] In the Dropout layer, some neurons of the feature vector are randomly discarded with a pre-set Dropout probability to reduce the risk of overfitting. For example, the Dropout probability is 0.5. In each forward propagation, a random mask of the same size as the input feature vector is generated, in which each element is determined to be zero based on the Dropout probability. The random mask is multiplied by the input feature vector to implement the operation of randomly discarding neurons.
[0062] A fully connected layer is used to map the hidden state of the text sequence output by Dropout after the Bi-LSTM layer to a dimensional space with the same number of entity categories, and a prediction vector with the same dimension as the number of entity categories is obtained. The prediction vector can be the probability distribution of each entity category obtained by the softmax function, and each dimension corresponds to the probability of an entity category.
[0063] The CRF layer is used to perform annotation prediction on the prediction vector of the text sequence output by the fully connected layer. For example, transition features and transition scores are defined in the CRF layer. These features and scores can be based on the transition probability between entity annotations, and the scores of all possible annotation sequences are calculated. These scores are determined by the transition features and transition scores. The Viterbi algorithm is used to find the annotation sequence with the highest probability in the CRF layer, and finally a normalized probability distribution is output, which represents the probability of the annotation sequence over the entire sequence.
[0064] The loss of the CRF layer is determined based on the predicted label sequence and the true label sequence. For example, a loss function may be used to determine the loss of the CRF layer. The loss function may be a negative log likelihood loss function (Negative Log Likelihood Loss), and the calculation formula thereof is:
[0065]
[0066] Among them, N is the size of a batch, x i is the input sequence of the i-th sample, y i is the corresponding true annotation sequence, θ is the model parameter, and P(y|x; θ) is the conditional probability of the annotation label y given the input sequence x and the model parameter θ.
[0067] The gradient of the loss with respect to each model parameter is calculated according to the loss function through the back-propagation algorithm. In an optional embodiment, before starting to calculate the gradient, the gradients of all parameters in the model are initialized to zero (for example, the gradient vector is set to a zero vector), starting from the output layer CRF, the error is back-propagated using the chain rule, that is, the gradient of the loss with respect to the output layer activation is calculated, and then these gradients are back-propagated layer by layer. For each layer of the model, the gradient of the loss with respect to the parameters of that layer is calculated, and the calculated gradients are accumulated into the previously initialized gradient vector. At each layer, the chain rule is used to calculate the gradient of the loss with respect to the input of the previous layer. Finally, each parameter is updated using gradient descent or its variant based on the accumulated gradient and the preset learning rate.
[0068] Repeat the steps of forward propagation, loss calculation, backpropagation, and parameter update until the performance of the model on the sample sequence no longer improves or reaches a predetermined number of iterations.
[0069] In order to help the model perform better when facing out-of-distribution (OOD) samples and enhance the generalization ability of the model, in the process of adjusting the model parameters of the initial entity recognition model according to the loss function, the following process can also be performed: determine the gradient of the model parameters of the entity recognition model according to the loss function; use an adversarial learning algorithm to process the sample sequence to obtain an adversarial sample; determine the first adversarial gradient based on the gradient generated by the adversarial sample and the gradient of the model parameters of the initial entity recognition model; use adaptive gradient clipping and gradient centering components to clip and center the first adversarial gradient to obtain a second adversarial gradient; update the model parameters of the initial entity recognition model according to the second adversarial gradient to obtain an entity recognition model.
[0070] In an optional embodiment, after calculating the gradient of the loss relative to each model parameter according to the negative log-likelihood loss function through the back-propagation algorithm, an adversarial learning algorithm is introduced for further processing. The adversarial learning algorithm may be, for example, a Fast Gradient Sign Method (FGSM), which generates adversarial samples by adding a perturbation with the same sign as the gradient to the original sample sequence. The perturbation (r) may be calculated by the L2 normalized calculation formula:
[0071] r=ε×g / ||g|| 2
[0072] Where ε is the perturbation parameter, g is the raw gradient of the embedding vector matrix, ||g|| 2 is the L2 norm of the original gradient.
[0073] For the generated adversarial sample, the back-propagation algorithm is used again to calculate the gradient of the loss with respect to the model parameters, and this gradient is accumulated to the original gradient of the model parameters to generate the first adversarial gradient.
[0074] Adaptive Gradient Clipping is applied to the first adversarial gradient, that is, a dynamic threshold is determined according to the current gradient and the gradient portion exceeding the threshold is clipped. In an optional embodiment, the following formula can be used for clipping:
[0075]
[0076] Among them, g t is the gradient of t iterations, ε and Γ are initialization parameter constants set to avoid freezing zeros, r is the dimension being processed, and θ is the model parameter of t iterations.
[0077] While or after the first adversarial gradient is adaptively clipped, the first adversarial gradient is centralized. For example, the gradient is zero-meaned. The calculation formula is:
[0078] g t =▽L t (θ t-1 )-mean(▽L t (θ t-1 ))
[0079] Among them, g t is the gradient of the t-round iteration, ▽ is the partial derivative, L t is the loss function for t rounds of iterations, θ t-1 are the model parameters for round t-1.
[0080] By adaptively clipping and centering the first adversarial gradient, the robustness and efficiency of adversarial attacks can be improved, and the model's defense against potential adversarial attacks can be enhanced. Updating the model parameters based on the generated second adversarial gradient can make the entity recognition model more robust and reduce the impact of adversarial perturbations. The model not only learns the characteristics of normal data, but also learns the patterns of adversarial perturbations, which helps to improve the model's generalization ability to different types of inputs.
[0081] In some embodiments of the present application, after the first adversarial gradient is clipped and centralized using adaptive gradient clipping and gradient centralization components, a momentum vector technology can also be introduced to accelerate the convergence speed of the gradient descent algorithm while reducing oscillations near the local minimum. The method is implemented through the following process: updating the current momentum vector according to the obtained parity of the current iteration round; when the current iteration round is an odd number, updating the current momentum vector according to the moment estimated average of the current momentum vector, and assigning a positive weight to the current momentum vector; when the current iteration round is an even number, updating the current momentum vector according to the moment estimated average of the previous momentum vector, and assigning a negative weight to the current momentum vector; adjusting the model parameters of the initial entity recognition model according to the updated current momentum vector.
[0082] In an optional embodiment, the momentum vector is used to store the weighted average of the previous gradient, so that the current update depends not only on the current gradient, but also on the historical information of the gradient. At the beginning of each iteration, check whether the number of current iterations is odd or even. For odd iterations, update the current momentum vector according to the moment estimated average of the current momentum vector (i.e., the weighted average of the current and past gradients), and assign a positive weight to it, that is, use a larger momentum coefficient to increase the impact of momentum. For even iterations, update the momentum vector according to the moment estimated average of the previous momentum vector, and assign a negative weight to it, that is, use a smaller momentum coefficient or reduce the momentum vector. Use the updated momentum vector to adjust the model parameters, reduce the shock during training, and make the parameter update more stable.
[0083] When updating the model parameters, in order to enhance the generalization ability of the model, improve the stability and robustness of the model, and improve the long-term performance of the model, when updating the model parameters of the initial entity recognition model according to the second adversarial gradient, the following process can also be performed: the parameter weights of the model parameters of the initial entity recognition model are adjusted using an optimization strategy, wherein the optimization strategy includes at least one of the following: L2 regularization of the parameter weights using the L2 norm, clipping the parameter weights according to a preset ratio, and resetting the exponential moving average of the parameters to the current parameter value according to a preset step size; and updating the model parameters of the initial entity recognition model according to the adjusted parameter weights.
[0084] In an optional embodiment, L2 regularization uses the L2 norm to regularize parameter weights to prevent model overfitting and encourage the model to learn a more compact weight representation by penalizing large weight values. For example, an L2 regularization term is added to the loss function, and the gradient of the regularized loss including the regularization term relative to the model parameters is calculated, and the gradient descent method is used to update the model parameters according to the calculated gradient.
[0085] Trimming parameter weights is used to limit the complexity of the model and prevent overfitting. For example, the trimming ratio is set before training. After each iteration, the model parameters are updated. For each parameter set (such as the weights of each layer), a trimming threshold is determined according to the trimming ratio. The parameters are sorted by their absolute values, and the top parameters that need to be trimmed are selected based on the preset ratio. The trimming operation is applied to these parameters to scale them to a size that does not exceed the trimming threshold. Finally, the trimmed parameter values are written back to the model weights.
[0086] Exponential Moving Average (EMA) is a method for calculating average values. EMA is usually used to smooth model parameters and reduce noise during training. In an optional embodiment, the exponential moving average of the parameters is reset to the current parameter value, that is, after each parameter update, the EMA of the parameter is updated. After reaching a preset step size of iterations, the EMA weight is directly updated to the current weight (i.e., resetting the EMA) to ensure that the model parameters do not deviate too far.
[0087] In order to facilitate the model to process the sample sequence and save computing resources, after obtaining the sample sequence, the following process can also be performed: truncate or pad the text sequence in the sample sequence according to the preset maximum sequence length; truncate or pad the annotation sequence in the sample sequence according to the maximum sequence length, wherein the length of the text sequence after truncation or padding is consistent with that of the annotation sequence after truncation or padding.
[0088] Figure 3 is a flow chart of a method for training an entity recognition model according to an embodiment of the present application, such as Figure 3 As shown, the method includes:
[0089] Step S301: Obtain a sample sequence, wherein the sample sequence includes a text sequence and a real annotation sequence corresponding to a named entity in the text sequence.
[0090] Step S302, using the embedded text vector layer of the initial entity recognition model to vectorize the text sequence to obtain an embedded vector sequence of the sample sequence.
[0091] Step S303, using the bidirectional long short-term memory network layer in the entity recognition model to process the embedding vector sequence to obtain a feature vector sequence including a forward hidden state and a reverse hidden state.
[0092] Step S304: using the discard layer of the entity recognition model to randomly discard some elements in the feature vector sequence according to a preset probability, to obtain a sparsely activated feature vector sequence.
[0093] Step S305, using the fully connected layer in the entity recognition model to perform feature integration on the feature vectors in the sparsely activated feature vector sequence to obtain a prediction vector sequence of fixed dimension.
[0094] Step S306: Use the conditional random field layer of the entity recognition model to perform label prediction on the named entities in the prediction vector sequence to obtain a prediction sample sequence.
[0095] Step S307, determining a loss function according to the predicted sample sequence and the true annotation sequence, and adjusting the model parameters of the initial entity recognition model according to the loss function.
[0096] Since the entity recognition model used in the above-mentioned named entity recognition method is trained in this embodiment, the overlapping contents will not be explained in detail, and the corresponding detailed explanations in the above-mentioned contents may be referred to.
[0097] Through the above steps S301 to S307, after obtaining the sample sequence, the embedded text vector layer of the initial entity recognition model is used to encode the sample sequence to obtain an embedded vector sequence, and then the bidirectional long short-term memory network layer is used to obtain a feature vector sequence containing a forward hidden state and a reverse hidden state, and the discard layer is used to randomly discard some elements in the feature vector sequence according to a preset probability to obtain a sparsely activated feature vector sequence, and then the fully connected layer is used to integrate features to obtain a fixed-dimensional prediction vector sequence, and the conditional random field layer is used for annotation prediction. Finally, the model parameters are adjusted according to the loss function to calculate the loss between the predicted annotation and the actual annotation. After continuous iterative training, the model parameters will be continuously optimized, and the recognition accuracy of the entity recognition model can be improved. This solves the technical problem that the generalization ability of the model is limited due to the imperfect training mechanism of the named entity recognition model in the related technology.
[0098] In some embodiments of the present application, the model parameters of the initial entity recognition model are adjusted according to the loss function, including: determining the gradient of the model parameters of the entity recognition model according to the loss function; processing the sample sequence using an adversarial learning algorithm to obtain an adversarial sample; determining a first adversarial gradient based on the gradient generated by the adversarial sample and the gradient of the model parameters of the initial entity recognition model; using adaptive gradient clipping and gradient centering components to clip and center the first adversarial gradient to obtain a second adversarial gradient; updating the model parameters of the initial entity recognition model according to the second adversarial gradient to obtain an entity recognition model.
[0099] It should be noted that Figure 3 The preferred implementation of the illustrated embodiment can be found in Figure 2 The corresponding solutions in the illustrated embodiment will not be described in detail here.
[0100] In order to facilitate the understanding of the above process, Figure 4 And a specific embodiment is provided to explain the training method of the above entity recognition model.
[0101] Figure 4 1 is an overall framework diagram of a method for implementing a training method for an entity recognition model according to an embodiment of the present application. Figure 4 As shown, in an optional embodiment, the following steps may be performed to train the entity recognition model:
[0102] Step 1: Preprocess the sample sequence to convert the text sequence into a form that is easier for the model to understand, including: setting the vocabulary, building a mapping dictionary corresponding to annotations and integer indices, and setting the maximum sequence length to truncate or pad the samples.
[0103] Step 2: After preprocessing, a multi-layer neural network learning model (i.e., named entity recognition model) is built based on the pre-trained model of Hugging Face, including: setting the parameters of the initial model, such as the hidden layer dimension, the embedded text vector layer dimension, etc. After setting the parameters, the embedded text vector layer, Bi-LSTM layer, Dropout layer, fully connected layer, and CRF layer are built in sequence, where the embedded text vector layer is used as the input layer of the model and the CRF layer is used as the output layer of the model.
[0104] Step 3: Train the built named entity recognition model, use the adversarial learning of Fast GradientMethod, and use the multi-component momentum adaptive dual average gradient optimizer to train the model to determine the optimal model, including: in each training batch, set the target loss function, such as the negative log-likelihood loss function, etc., add adversarial learning after the neural network obtains the gradient through back propagation, determine the adversarial gradient after disturbance, and add multiple components such as adaptive gradient clipping, gradient centering, and positive and negative momentum for optimization training to determine the optimal model.
[0105] Step 4: Load the optimal model determined by training for named entity recognition, set the model to eval mode, turn off automatic gradient calculation, obtain the recognition results and organize them to meet the needs of related natural language processing tasks such as information extraction, and ensure the availability of the recognition results.
[0106] Step 5: In addition, the entity recognition model of the present application and the entity recognition model in the related technology can be trained using the same training data set, and the recognition results can be compared to intuitively reflect the improvement in recognition performance such as recognition accuracy between the model in the present application that adds adversarial learning and the model in the related technology.
[0107] In a specific embodiment, intelligent information extraction is performed on the product information text data crawled from an e-commerce website published by an e-commerce company to form a standardized document to facilitate subsequent reading and further data analysis for business decision-making. An example of obtaining the original e-commerce product information text is as follows (the specific product brand and model are blurred in this manual):
[0108] "Brand A Model a1, Model a2, Model a3, Model a4 In-ear heavy bass earphones Earbuds with microphone Brand B Original Model b1 Lens Model b2, is, stm, Model b3, Model b4, Model b5 SLR 18-55 Brand C Men's watch Swiss Brand C Observatory certified watch Model c1 Brand D Compound aromatherapy 10ml, Model d1 German o home, official genuine authorization, Model e1 authentic Brand E switching power supply, Model e2, 5v / 10a Brand F Computer room dedicated air conditioner Model f1, 17kw, computer room precision air conditioner constant temperature and humidity upper air supply Brand G Watch Geographic Master Series Model g1 Swiss imported automatic men's watch Model g2 New, Brand H Model h1 Universal ink cartridge for fountain pens, replaceable ink cartridge, erasable blue, red, 20 pieces in a box 3m American Brand I Model i1 Transparent hundred grid test tape width 19mm* length 33m environmentally friendly removable Brand J matching battery Model j1". When performing named entity recognition on the above original e-commerce product information text, the following process can be used:
[0109] Step 1: Preprocess the original product information text data. Analyze the original information text, define the Chinese named entities for annotation as product category, product brand, product model, and other product descriptions, and set them as HCCX, HPPX, XH, and MISC annotation tags respectively. The definition vocabulary is as follows:
[0110] VOCAB=(' <pad>','[CLS]','0','B-MISC','B-HPPX','B-XH','B-HCCX','I-MISC','I-HPPX','I-XH','I-HCCX').
[0111] According to this vocabulary, the annotation labels are mapped to integer indices to form a mapping dictionary for this information extraction annotation task, and the maximum sequence length processed by the model is set to 254.
[0112] Step 2: Build the model. Set the embedded text vector dimension to 768, the model hidden layer dimension to 256, set the underlying representation model to "chinese-roberta-wwm-ext", set the input data dimension of the Bi-LSTM layer to the embedded text vector dimension of 768, the model Dropout probability to 0.1, the dimension of the fully connected layer to the hidden layer dimension of 256, and use the CRF layer as the output layer of the overall hybrid learning model.
[0113] Step 3: Model training. On an AI platform, the GPU of TESLA V100 is used for training. The training generation is set to 30, the size of each batch is 64, and 10% of the training data is used as warm-up, that is, on 10% of the data, the model is trained with a smaller learning rate. After the training of 10% of the data is completed, the learning rate is adjusted back to the normal level, and the L2 normalized perturbation is used as the adversarial gradient. At the same time, the momentum-based adaptive dual average gradient optimizer with multiple components such as adaptive gradient clipping, gradient centering, positive and negative momentum, weight soft regularization and stable decay, and weight moving average is used as a convergence tool to find the optimal model. The model with the smallest loss function and the highest accuracy on the verification data is selected as the optimal model.
[0114] Step 4: Named entity inference and information extraction results. Load the optimal model obtained from the above training, input the original e-commerce product information, perform Chinese named entity annotation, and automatically standardize the inferred and annotated results according to the labels. Some of the results are shown in the following table:
[0115]
[0116] From the results, it can be seen that the text information crawled from the e-commerce website was labeled and identified, and key information such as product type, product brand, product model, and other product descriptions were successfully identified, forming intelligent and standardized data that can be further decided and analyzed.
[0117] In another specific embodiment, in a company's automated office system, intelligent information extraction is performed on leadership appointment and dismissal information data to form efficient documents, thereby helping company decision makers to quickly obtain valuable information.
[0118] Step 1: Preprocess the original information text data. Read the original document to form the original information json text, analyze the original information text, define the annotated Chinese named entities as name, employment unit, employment position, dismissal unit, dismissal position, set them as XM, RZDW, RZZW, MZDW, MZZW respectively, and define the vocabulary as follows:
[0119] VOCAB=(' <pad>','[CLS]','[SEP]','0','B-MZZW','B-MZDW','B-RZDW','B-RZZW','B-XM','I-MZZW','I-MZDW','I-RZDW','I-RZZW','I-XM').
[0120] According to this vocabulary, the annotation labels are mapped to integer indices to form a mapping dictionary for this information extraction annotation task, and the maximum sequence length processed by the model is set to 254.
[0121] Step 2: Build the model. Set the embedded text vector dimension to 768, the model hidden layer dimension to 256, set the underlying representation model to "chinese-macbert-base", set the input data dimension of the Bi-LSTM layer to the embedded text vector dimension of 768, the model Dropout probability to 0.1, the dimension of the fully connected layer to the hidden layer dimension of 256, and use the CRF layer as the output layer of the overall hybrid learning model.
[0122] Step 3: Model training. On an AI platform, the GPU of TESLA V100 is used for training. The training generation is set to 30, the size of each batch is 64, 10% of the training data is used as warm-up, and L2 normalized perturbation is used as the adversarial gradient. At the same time, the momentum-based adaptive dual average gradient optimizer with multiple components such as adaptive gradient clipping, gradient centering, positive and negative momentum, weight soft regularization and stable decay, and weight moving average is used as a convergence tool to find the optimal model. The model with the smallest loss function and the highest accuracy on the verification data is selected as the optimal model.
[0123] Step 4: Named entity inference and information extraction results. Load the optimal model obtained from the above training, input the original information json text information, perform Chinese named entity annotation, and automatically organize the inferred and annotated results. Some of the results are as follows:
[0124]
[0125] From the results, it can be seen that the text information in the issued documents was annotated and identified, and key information such as name, employment unit, employment position, dismissal unit, dismissal position, etc. were successfully identified, forming intelligent and standardized data, forming documents that can be read efficiently, and improving office efficiency.
[0126] In addition, in the embodiments of the present application, the authoritative Chinese public data sets MSRA (Chinese named entity recognition data set in the news field of Microsoft Research Asia) and CKKS (Chinese medical record named entity recognition data set of the China Knowledge Graph and Semantic Computing Conference) are used for comparison with the original Bi-LSTM-CRF method, and the accuracy (precision), recall (recall) and the harmonic average f1 value (f1-score) of the accuracy and recall of each labeled recognition subclass are compared, as well as the overall micro average (micro avg), macro average (macro avg), and weighted average (weighted avg). The results are shown in Tables 1 to 4 respectively:
[0127] Table 1: Recognition performance of the original method on the MSRA dataset.
[0128]
[0129]
[0130] Table 2: Recognition performance of this method on the MSRA dataset.
[0131]
[0132] It can be seen that on the MSRA dataset, the accuracy, recall, harmonic mean f1 value of accuracy and recall, overall micro-average, macro-average, and weighted average accuracy of this method in each annotation recognition subcategory task and the overall task are all above 90%. Compared with the original method, the recognition performance has been significantly improved.
[0133] Table 3: Recognition performance of the original method on the CKKS dataset.
[0134]
[0135] Table 4: Recognition performance of this method on the CKKS dataset.
[0136]
[0137] Comparing Table 3 with Table 4, we can see that on the CKKS dataset, this method also has a relatively good improvement in Chinese named entity recognition performance compared to the original method, especially in the harmonic mean f1 value of precision and recall, the overall micro-average, macro-average, and weighted average values.
[0138] According to an embodiment of the present application, a device for named entity recognition is provided. It should be noted that the device for named entity recognition in the embodiment of the present application can be used to execute the method for named entity recognition provided in the embodiment of the present application. The device for named entity recognition provided in the embodiment of the present application is introduced below.
[0139] Figure 5 is a structural diagram of a device for named entity recognition provided according to an embodiment of the present application. Figure 5 As shown, the device comprises:
[0140] A first acquisition module 50 is used to acquire a text sequence to be recognized;
[0141] A first conversion module 52 is used to perform vectorization processing on the text sequence to be recognized using the embedded text vector layer in the entity recognition model to obtain an embedded vector sequence;
[0142] The recognition module 54 is used to use the hidden layer in the entity recognition model to recognize the named entities in the embedded vector sequence to obtain the named entities of the text sequence to be recognized, wherein the entity recognition model is a model trained by adversarial learning.
[0143] Through the above-mentioned acquisition module 50, conversion module 52 and recognition module 54, the purpose of identifying named entities in text sequences is achieved. Since the generalization ability of the entity recognition model is significantly improved after adversarial training, the technical effect of being able to more accurately identify untrained text data is achieved, thereby solving the technical problem that the training mechanism of the named entity recognition model in the related technology is not perfect, resulting in limited generalization ability of the model when identifying untrained data or data that slightly deviates from the training distribution.
[0144] In the device for named entity recognition provided in an embodiment of the present application, the first acquisition module is also used to update the current momentum vector according to the parity of the current iteration round obtained; when the current iteration round is an odd number, the current momentum vector is updated according to the moment estimated average of the current momentum vector, and a positive weight is assigned to the current momentum vector; when the current iteration round is an even number, the current momentum vector is updated according to the moment estimated average of the previous momentum vector, and a negative weight is assigned to the current momentum vector; and the model parameters of the initial entity recognition model are adjusted according to the updated current momentum vector.
[0145] In the device for named entity recognition provided in an embodiment of the present application, the first acquisition module is also used to truncate or fill the text sequence in the sample sequence according to a preset maximum sequence length; truncate or fill the annotation sequence in the sample sequence according to the maximum sequence length, wherein the length of the text sequence after truncation or filling is consistent with the length of the annotation sequence after truncation or filling.
[0146] In the device for named entity recognition provided in an embodiment of the present application, the first conversion module is also used to use the bidirectional long short-term memory network layer in the entity recognition model to process the embedded vector sequence to obtain a feature vector sequence containing a forward hidden state and a reverse hidden state; use the discard layer of the entity recognition model to randomly discard some elements in the feature vector sequence according to a preset probability to obtain a sparsely activated feature vector sequence; use the fully connected layer in the entity recognition model to integrate the features of the feature vectors in the sparsely activated feature vector sequence to obtain a prediction vector sequence of fixed dimension; use the conditional random field layer of the entity recognition model to perform label prediction on the named entities in the prediction vector sequence to obtain the named entities of the text sequence to be recognized.
[0147] In the device for named entity recognition provided in an embodiment of the present application, the first conversion module is also used to obtain a sample sequence, wherein the sample sequence includes a text sequence and a real annotation sequence corresponding to the named entities in the text sequence; the text sequence is vectorized using the embedded text vector layer of the initial entity recognition model to obtain an embedded vector sequence of the sample sequence; the embedded vector sequence is processed using the bidirectional long short-term memory network layer in the entity recognition model to obtain a feature vector sequence containing a forward hidden state and a reverse hidden state; the discard layer of the entity recognition model is used to randomly discard some elements in the feature vector sequence according to a preset probability to obtain a sparsely activated feature vector sequence; the fully connected layer in the entity recognition model is used to integrate the features of the feature vectors in the sparsely activated feature vector sequence to obtain a prediction vector sequence of fixed dimension; the conditional random field layer of the entity recognition model is used to perform annotation prediction on the named entities in the prediction vector sequence to obtain a prediction sample sequence; the loss function is determined based on the predicted sample sequence and the real annotation sequence, and the model parameters of the initial entity recognition model are adjusted according to the loss function.
[0148] In the device for named entity recognition provided in an embodiment of the present application, the first conversion module is also used to adjust the parameter weights of the model parameters of the initial entity recognition model using an optimization strategy, wherein the optimization strategy includes at least one of the following: using the L2 norm to perform L2 regularization on the parameter weights, trimming the parameter weights according to a preset ratio, and resetting the exponential moving average of the parameters to the current parameter value according to a preset step size; and updating the model parameters of the initial entity recognition model according to the adjusted parameter weights.
[0149] In the device for named entity recognition provided in an embodiment of the present application, the recognition module is also used to determine the gradient of the model parameters of the entity recognition model based on the loss function; use an adversarial learning algorithm to process the sample sequence to obtain an adversarial sample; determine a first adversarial gradient based on the gradient generated by the adversarial sample and the gradient of the model parameters of the initial entity recognition model; use adaptive gradient clipping and gradient centering components to clip and center the first adversarial gradient to obtain a second adversarial gradient; update the model parameters of the initial entity recognition model according to the second adversarial gradient to obtain an entity recognition model.
[0150] In the device for named entity recognition provided in an embodiment of the present application, the recognition module is also used to obtain a sample sequence, wherein the sample sequence includes a text sequence and a real annotation sequence corresponding to the named entities in the text sequence; the text sequence is vectorized using the embedded text vector layer of the initial entity recognition model to obtain an embedded vector sequence of the sample sequence; the embedded vector sequence is processed using the bidirectional long short-term memory network layer in the entity recognition model to obtain a feature vector sequence containing a forward hidden state and a reverse hidden state; the discard layer of the entity recognition model is used to randomly discard some elements in the feature vector sequence according to a preset probability to obtain a sparsely activated feature vector sequence; the fully connected layer in the entity recognition model is used to integrate the features of the feature vectors in the sparsely activated feature vector sequence to obtain a prediction vector sequence of fixed dimension; the conditional random field layer of the entity recognition model is used to perform annotation prediction on the named entities in the prediction vector sequence to obtain a prediction sample sequence; the loss function is determined based on the predicted sample sequence and the real annotation sequence, and the model parameters of the initial entity recognition model are adjusted according to the loss function.
[0151] In the device for named entity recognition provided in an embodiment of the present application, the recognition module is also used to determine the gradient of the model parameters of the entity recognition model based on the loss function; use an adversarial learning algorithm to process the sample sequence to obtain an adversarial sample; determine a first adversarial gradient based on the gradient generated by the adversarial sample and the gradient of the model parameters of the initial entity recognition model; use adaptive gradient clipping and gradient centering components to clip and center the first adversarial gradient to obtain a second adversarial gradient; update the model parameters of the initial entity recognition model according to the second adversarial gradient to obtain an entity recognition model.
[0152] Figure 6 is a structural diagram of an entity recognition model training device provided according to an embodiment of the present application. Figure 6 As shown, the device comprises:
[0153] The second acquisition module 60 is used to acquire a sample sequence, wherein the sample sequence includes a text sequence and a real annotation sequence corresponding to a named entity in the text sequence.
[0154] The second conversion module 62 is used to use the embedded text vector layer in the entity recognition model to vectorize the text sequence to be recognized to obtain an embedded vector sequence; use the bidirectional long short-term memory network layer in the entity recognition model to process the embedded vector sequence to obtain a feature vector sequence containing a forward hidden state and a reverse hidden state; use the discard layer of the entity recognition model to randomly discard some elements in the feature vector sequence according to a preset probability to obtain a sparsely activated feature vector sequence; use the fully connected layer in the entity recognition model to integrate the features of the feature vectors in the sparsely activated feature vector sequence to obtain a fixed-dimensional prediction vector sequence; use the conditional random field layer of the entity recognition model to label and predict the named entities in the prediction vector sequence to obtain a prediction sample sequence.
[0155] The adjustment module 64 is used to determine a loss function according to the predicted sample sequence and the true annotation sequence, and adjust the model parameters of the initial entity recognition model according to the loss function.
[0156] An embodiment of the present application also provides an electronic device, which includes a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory, and is used to execute program instructions to implement the following functions: obtaining a text sequence to be recognized; using an embedded text vector layer in an entity recognition model to vectorize the text sequence to be recognized, and obtain an embedded vector sequence; using a hidden layer in the entity recognition model to recognize named entities in the embedded vector sequence, and obtain named entities in the text sequence to be recognized, wherein the entity recognition model is a model trained using an adversarial learning method.
[0157] An embodiment of the present application also provides a non-volatile storage medium, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned method of named entity recognition by running the computer program.
[0158] An embodiment of the present application also provides a computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for named entity recognition in each embodiment of the present application are implemented.
[0159] An embodiment of the present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the method for named entity recognition in each embodiment of the present application.
[0160] The embodiments of the present application also provide a computer program, which, when executed by a processor, implements the steps of the method for named entity recognition in each embodiment of the present application.
[0161] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.
[0162] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.
[0163] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0164] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0165] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0166] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.
[0167] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.< / pad> < / pad>
Claims
1. A method for named entity recognition, characterized in that: include: Get the text sequence to be recognized; The embedded text vector layer in the entity recognition model is used to vectorize the text sequence to be recognized to obtain an embedded vector sequence; The named entities in the embedded vector sequence are identified by using a hidden layer in the entity recognition model to obtain the named entities of the text sequence to be identified, wherein the entity recognition model is a model trained by using an adversarial learning method.
2. The method according to claim 1, characterized in that Using the hidden layer in the entity recognition model to recognize the named entities in the embedding vector sequence to obtain the named entities of the text sequence to be recognized, including: The embedding vector sequence is processed using a bidirectional long short-term memory network layer in the entity recognition model to obtain a feature vector sequence including a forward hidden state and a reverse hidden state; Using the discard layer of the entity recognition model to randomly discard some elements in the feature vector sequence according to a preset probability, so as to obtain a sparsely activated feature vector sequence; Using a fully connected layer in the entity recognition model to perform feature integration on the feature vectors in the sparsely activated feature vector sequence to obtain a prediction vector sequence of fixed dimension; The conditional random field layer of the entity recognition model is used to perform annotation prediction on the named entities in the prediction vector sequence to obtain the named entities of the text sequence to be recognized.
3. The method according to claim 1, characterized in that The entity recognition model is trained in the following way: Acquire a sample sequence, wherein the sample sequence includes a text sequence and a real annotation sequence corresponding to a named entity in the text sequence; Using the embedded text vector layer of the initial entity recognition model to vectorize the text sequence to obtain an embedded vector sequence of the sample sequence; The embedding vector sequence is processed using a bidirectional long short-term memory network layer in the entity recognition model to obtain a feature vector sequence including a forward hidden state and a reverse hidden state; Using the discard layer of the entity recognition model to randomly discard some elements in the feature vector sequence according to a preset probability, so as to obtain a sparsely activated feature vector sequence; Using a fully connected layer in the entity recognition model to perform feature integration on the feature vectors in the sparsely activated feature vector sequence to obtain a prediction vector sequence of fixed dimension; Using the conditional random field layer of the entity recognition model to perform annotation prediction on the named entities in the prediction vector sequence to obtain a prediction sample sequence; A loss function is determined according to the predicted sample sequence and the true annotation sequence, and model parameters of the initial entity recognition model are adjusted according to the loss function.
4. The method according to claim 3, characterized in that Adjusting the model parameters of the initial entity recognition model according to the loss function includes: Determining the gradient of the model parameters of the entity recognition model according to the loss function; Using an adversarial learning algorithm to process the sample sequence to obtain an adversarial sample; Determine a first adversarial gradient according to the gradient generated by the adversarial sample and the gradient of the model parameters of the initial entity recognition model; Adopting an adaptive gradient clipping and gradient centralization component to clip and centralize the first adversarial gradient to obtain a second adversarial gradient; The model parameters of the initial entity recognition model are updated according to the second adversarial gradient to obtain the entity recognition model.
5. The method according to claim 4, characterized in that After the first adversarial gradient is clipped and centralized using the adaptive gradient clipping and gradient centralization components, the method further includes: Update the current momentum vector according to the obtained parity of the current iteration round; When the current iteration round is an odd number, updating the current momentum vector according to the moment estimated average value of the current momentum vector, and assigning a positive weight to the current momentum vector; When the current iteration round is an even number, updating the current momentum vector according to the moment estimated average value of the previous momentum vector, and assigning a negative weight to the current momentum vector; The model parameters of the initial entity recognition model are adjusted according to the updated current momentum vector.
6. The method according to claim 4, characterized in that Updating the model parameters of the initial entity recognition model according to the second adversarial gradient includes: The parameter weights of the model parameters of the initial entity recognition model are adjusted by using an optimization strategy, wherein the optimization strategy includes at least one of the following: L2 regularization of the parameter weights by using an L2 norm, clipping the parameter weights according to a preset ratio, and resetting the exponential moving average of the parameters to the current parameter value according to a preset step size; The model parameters of the initial entity recognition model are updated according to the adjusted parameter weights.
7. The method according to claim 3, characterized in that After acquiring the sample sequence, the method further includes: Truncating or padding the text sequence in the sample sequence according to a preset maximum sequence length; The tag sequence in the sample sequence is truncated or padded according to the maximum sequence length, wherein the length of the text sequence after truncating or padded is consistent with the length of the tag sequence after truncating or padded.
8. A method for training an entity recognition model, characterized in that: include: Acquire a sample sequence, wherein the sample sequence includes a text sequence and a real annotation sequence corresponding to a named entity in the text sequence; Using the embedded text vector layer of the initial entity recognition model to vectorize the text sequence to obtain an embedded vector sequence of the sample sequence; The embedding vector sequence is processed using a bidirectional long short-term memory network layer in the entity recognition model to obtain a feature vector sequence including a forward hidden state and a reverse hidden state; Using the discard layer of the entity recognition model to randomly discard some elements in the feature vector sequence according to a preset probability, so as to obtain a sparsely activated feature vector sequence; Using a fully connected layer in the entity recognition model to perform feature integration on the feature vectors in the sparsely activated feature vector sequence to obtain a prediction vector sequence of fixed dimension; Using the conditional random field layer of the entity recognition model to perform annotation prediction on the named entities in the prediction vector sequence to obtain a prediction sample sequence; A loss function is determined according to the predicted sample sequence and the true annotation sequence, and model parameters of the initial entity recognition model are adjusted according to the loss function.
9. The method according to claim 8, characterized in that Adjusting the model parameters of the initial entity recognition model according to the loss function includes: Determining the gradient of the model parameters of the entity recognition model according to the loss function; Using an adversarial learning algorithm to process the sample sequence to obtain an adversarial sample; Determine a first adversarial gradient according to the gradient generated by the adversarial sample and the gradient of the model parameters of the initial entity recognition model; Adopting an adaptive gradient clipping and gradient centralization component to clip and centralize the first adversarial gradient to obtain a second adversarial gradient; The model parameters of the initial entity recognition model are updated according to the second adversarial gradient to obtain the entity recognition model.
10. A device for named entity recognition, characterized in that: include: A first acquisition module is used to acquire a text sequence to be recognized; A first conversion module is used to use the embedded text vector layer in the entity recognition model to vectorize the text sequence to be recognized to obtain an embedded vector sequence; A recognition module is used to use the hidden layer in the entity recognition model to recognize the named entities in the embedded vector sequence to obtain the named entities of the text sequence to be recognized, wherein the entity recognition model is an entity recognition model trained by adversarial learning.
11. A device for training an entity recognition model, characterized in that: include: A second acquisition module is used to acquire a sample sequence, wherein the sample sequence includes a text sequence and a real annotation sequence corresponding to a named entity in the text sequence; The second conversion module is used to use the embedded text vector layer in the entity recognition model to vectorize the text sequence to be recognized to obtain an embedded vector sequence; use the bidirectional long short-term memory network layer in the entity recognition model to process the embedded vector sequence to obtain a feature vector sequence containing a forward hidden state and a reverse hidden state; use the discard layer of the entity recognition model to randomly discard some elements in the feature vector sequence according to a preset probability to obtain a sparsely activated feature vector sequence; use the fully connected layer in the entity recognition model to integrate the features of the feature vectors in the sparsely activated feature vector sequence to obtain a fixed-dimensional prediction vector sequence; use the conditional random field layer of the entity recognition model to perform annotation prediction on the named entities in the prediction vector sequence to obtain a prediction sample sequence; The adjustment module is used to determine a loss function according to the predicted sample sequence and the real annotation sequence, and adjust the model parameters of the initial entity recognition model according to the loss function.
12. An electronic device, characterized in that: include: A memory for storing program instructions; A processor is connected to the memory and is used to execute program instructions to implement the following functions: obtaining a text sequence to be recognized; using an embedded text vector layer in an entity recognition model to vectorize the text sequence to be recognized to obtain an embedded vector sequence; using a hidden layer in the entity recognition model to recognize named entities in the embedded vector sequence to obtain named entities in the text sequence to be recognized, wherein the entity recognition model is an entity recognition model trained by adversarial learning.
13. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the method for named entity recognition described in any one of claims 1 to 7 or the method for training an entity recognition model described in any one of claims 8 to 9 by running the computer program.
14. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the method for named entity recognition described in any one of claims 1 to 7 or the method for training an entity recognition model described in any one of claims 8 to 9 is implemented.
Citation Information
Cited By
Ear-nose-throat nursing scheme recommendation method based on knowledge graph
CN119207778A
