Entity recognition method and device based on attention Bi-LSTM+CRF model

Abstract

The application relates to an entity recognition method and device based on an attention-based Bi-LSTM+CRF model. The method comprises the following steps: preprocessing a text to be recognized, to obtain an input vector of the text to be recognized, wherein the input vector comprises a character granularity vector, a word granularity vector and a vocabulary granularity vector; inputting the input vector into an attention-based Bi-LSTM+CRF model; adopting an attention mechanism in an input feature layer of the attention-based Bi-LSTM+CRF model, to respectively assign weights to vectors of different granularities at different positions of the input vector, and to obtain a mixed vector after weighted sum operation; and obtaining an entity recognition result output by the attention-based Bi-LSTM+CRF model to be trained according to the mixed vector. The scheme provided in the application can accurately recognize entities in a text, and achieves the best entity recognition effect.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to an entity recognition method and apparatus based on an attention-based Bi-LSTM+CRF model. Background Technology

[0002] Entity recognition is a common problem in the field of natural language processing, mainly involving extracting target words from a text. LSTM (Long-Short Term Memory) is an improved version of RNN (Recurrent Neural Network). As a representative of sequence modeling, LSTM is naturally suitable for sequence modeling. Although it adds gate mechanisms such as input gate and forget gate compared to RNN, it is still insufficient for modeling long texts with complex patterns and cannot capture long-distance text information.

[0003] Based on the characteristics of LSTM, the related technologies Bi-LSTM (Bi-directional Long-Short Term Memory) + CRF (Conditional Random Field, a discriminative probabilistic graphical model) model obtains input features in a relatively coarse way by simply superimposing the character-level features and word-level features of the text. It performs well for the recognition of common simple text entities, but performs poorly for the recognition of long text entities with complex patterns.

[0004] In summary, the Bi-LSTM+CRF model of the relevant technologies is insufficient for modeling long texts, unable to capture long-distance text information, unable to accurately identify entities in long texts with complex patterns, and unable to achieve the best entity recognition results. Summary of the Invention

[0005] To address or partially address the problems existing in related technologies, this application provides an entity recognition method and apparatus based on an attention-based Bi-LSTM+CRF model, which can accurately recognize entities in text and achieve the best entity recognition effect.

[0006] The first aspect of this application provides an entity recognition method based on an attention-based Bi-LSTM+CRF model, the method comprising:

[0007] The training set is preprocessed to obtain the input vector of the training set, which includes character granularity vector, word granularity vector, and dictionary granularity vector;

[0008] The input vector is input into the attention-based Bi-LSTM+CRF model to be trained;

[0009] In the input feature layer of the attention-based Bi-LSTM+CRF model to be trained, an attention mechanism is used to assign weights to the character granular vector, word granular vector, and vocabulary granular vector at different positions of the input vector, respectively, to obtain a weighted sum of the mixed vectors.

[0010] Based on the hybrid vector, the entity recognition result output by the attention-based Bi-LSTM+CRF model to be trained is obtained;

[0011] Based on the entity recognition results and the training set, the loss function value is obtained;

[0012] If the loss function value is less than the set loss threshold and / or the number of iterative training iterations reaches the set number of iterations, the training of the attention-based Bi-LSTM+CRF model is considered complete.

[0013] The text to be identified is preprocessed to obtain the input vector of the text to be identified;

[0014] The input vector of the text to be identified is input into the trained attention-based Bi-LSTM+CRF model so that the attention-based Bi-LSTM+CRF model outputs the entity recognition result.

[0015] Preferably, the attention-based Bi-LSTM+CRF model includes an input feature layer, a first attention layer set before the forward LSTM, and a second attention layer set before the backward LSTM.

[0016] Preferably, in the input feature layer of the attention-based Bi-LSTM+CRF model to be trained, an attention mechanism is used to assign weights to the character-level vectors, word-level vectors, and vocabulary-level vectors at different positions of the input vector, respectively, to obtain a weighted sum of the resulting mixed vectors, including:

[0017] In the input feature layer of the attention-based Bi-LSTM+CRF model to be trained, an attention mechanism is used to assign weights to the character granular vector, word granular vector, and vocabulary granular vector at each different position of the input vector.

[0018] Based on the weights of the character-level vector, word-level vector, and dictionary-level vector at each different position in the input vector, an attention mechanism is used to perform a weighted sum operation on the character-level vector, word-level vector, and dictionary-level vector at each different position in the input vector, resulting in a weighted sum hybrid vector.

[0019] Preferably, obtaining the entity recognition result output by the attention-based Bi-LSTM+CRF model to be trained based on the mixture vector includes:

[0020] The mixed vector is input into the first attention layer, and an attention mechanism is used to obtain the first weighted sum vector output by the first attention layer;

[0021] The first weighted sum vector is input into the forward LSTM to obtain the forward fusion vector output by the forward LSTM;

[0022] The forward fusion vector is input into the second attention layer, and an attention mechanism is used to obtain the second weighted sum vector;

[0023] The second weighted sum vector is input into the backward LSTM to obtain the backward fusion vector output by the backward LSTM;

[0024] The backward fusion vector is input into the CRF layer to obtain the entity recognition result output by the CRF layer.

[0025] Preferably, the method further includes:

[0026] If the loss function value is greater than or equal to the set loss threshold and / or the number of iterations for training does not reach the set number of iterations, the following step is executed: in the input feature layer of the attention-based Bi-LSTM+CRF model to be trained, an attention mechanism is used to assign weights to the character granular vector, word granular vector, and dictionary granular vector at different positions of the input vector, respectively, to obtain a weighted sum of the mixed vectors.

[0027] A second aspect of this application provides an entity recognition device based on an attention-based Bi-LSTM+CRF model, the device comprising:

[0028] The preprocessing module is used to preprocess the training set to obtain the input vector of the training set, wherein the input vector includes character granular vector, word granular vector, and dictionary granular vector;

[0029] The vector input module is used to input the input vector obtained by the preprocessing module into the attention-based Bi-LSTM+CRF model to be trained;

[0030] The vector mixing module is used to assign weights to the character-granular vector, word-granular vector, and vocabulary-granular vector at different positions of the input vector input by the vector input module in the input feature layer of the attention-based Bi-LSTM+CRF model to be trained, using an attention mechanism, to obtain a weighted and mixed vector.

[0031] The result acquisition module is used to obtain the entity recognition result output by the attention-based Bi-LSTM+CRF model to be trained based on the mixed vector obtained by the vector mixing module.

[0032] The loss calculation module is used to obtain the loss function value based on the entity recognition result obtained by the result acquisition module and the training set;

[0033] The training completion module is used to determine that the training of the attention-based Bi-LSTM+CRF model is complete if the loss function value obtained by the loss calculation module is less than a set loss threshold and / or the number of iterative training cycles reaches a set number of iterations.

[0034] The preprocessing module is also used to preprocess the text to be identified, and obtain the input vector of the text to be identified.

[0035] The vector input module is further configured to input the input vector of the text to be identified obtained by the preprocessing module into the attention-based Bi-LSTM+CRF model after training, so that the attention-based Bi-LSTM+CRF model outputs the entity recognition result.

[0036] Preferably, the vector mixing module includes:

[0037] The weight assignment submodule is used to assign weights to the character-granular vector, word-granular vector, and vocabulary-granular vector at each different position of the input vector input by the vector input module in the input feature layer of the attention-based Bi-LSTM+CRF model to be trained, using an attention mechanism.

[0038] The vector operation submodule is used to assign weights to the character-granular vector, word-granular vector, and lexicon-granular vector at each different position of the input vector given by the submodule according to the weights. Using an attention mechanism, the submodule performs a weighted sum operation on the character-granular vector, word-granular vector, and lexicon-granular vector at each different position of the input vector input by the vector input module to obtain a weighted sum operation hybrid vector.

[0039] Preferably, the result acquisition module is further used for:

[0040] The mixed vector obtained by the vector mixing module is input into the first attention layer of the attention-based Bi-LSTM+CRF model to be trained, and the first weighted sum vector output by the first attention layer is obtained by using the attention mechanism.

[0041] The first weighted sum vector is input into the forward LSTM of the attention-based Bi-LSTM+CRF model to be trained, and the forward fusion vector output by the forward LSTM is obtained.

[0042] The forward fusion vector is input into the second attention layer of the attention-based Bi-LSTM+CRF model to be trained, and the second weighted sum vector is obtained by using the attention mechanism.

[0043] The second weighted sum vector is input into the backward LSTM of the attention-based Bi-LSTM+CRF model to be trained, and the backward fusion vector output by the backward LSTM is obtained.

[0044] The backward fusion vector is input into the CRF layer of the attention-based Bi-LSTM+CRF model to be trained, and the entity recognition result output by the CRF layer is obtained.

[0045] A third aspect of this application provides an electronic device, comprising:

[0046] Processor; and

[0047] A memory that stores executable code, which, when executed by the processor, causes the processor to perform the method described above.

[0048] A fourth aspect of this application provides a computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method described above.

[0049] The technical solution provided in this application may include the following beneficial effects:

[0050] The technical solution of this application includes three different granularities of input vectors: character-level vectors, word-level vectors, and vocabulary-level vectors. In the input feature layer of the attention-based Bi-LSTM+CRF model, an attention mechanism is used to assign weights to the three different granularities of vectors at different positions of the input vector, resulting in a weighted sum of the mixed vectors. Based on the mixed vector, the entity recognition result output by the attention-based Bi-LSTM+CRF model is obtained. This allows the attention-based Bi-LSTM+CRF model to dynamically give different weights to the different granularities of vectors at different positions of the input vector during the training process, dynamically adjusting its fitting ability to the input vector, comprehensively enhancing the information contained in the input vector, enriching the information at each position in the vector, and enabling better training of the attention-based Bi-LSTM+CRF model, improving the model's training efficiency, accurately recognizing entities in text, especially long texts, improving the accuracy and speed of entity recognition, and achieving the best entity recognition effect.

[0051] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0052] The above and other objects, features and advantages of this application will become more apparent from the more detailed description of exemplary embodiments thereof in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments thereof.

[0053] Figure 1 This is a flowchart illustrating the entity recognition method based on the attention-based Bi-LSTM+CRF model as shown in the embodiments of this application;

[0054] Figure 2 This is another schematic diagram of the entity recognition method based on the attention-based Bi-LSTM+CRF model shown in the embodiments of this application;

[0055] Figure 3 This is a schematic diagram of the input vector and weights of the entity recognition method based on the attention-based Bi-LSTM+CRF model shown in the embodiments of this application;

[0056] Figure 4 This is a structural block diagram of the attention-based Bi-LSTM+CRF model for entity recognition method based on the attention-based Bi-LSTM+CRF model shown in the embodiments of this application;

[0057] Figure 5 This is a schematic diagram of the structure of an entity recognition device based on an attention-based Bi-LSTM+CRF model, as shown in an embodiment of this application.

[0058] Figure 6 This is another schematic diagram of the entity recognition device based on the attention-based Bi-LSTM+CRF model shown in the embodiments of this application;

[0059] Figure 7 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application. Detailed Implementation

[0060] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.

[0061] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0062] It should be understood that although the terms "first," "second," "third," etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0063] This application provides an entity recognition method based on an attention-based Bi-LSTM+CRF model, which can accurately identify entities in text and achieve the best entity recognition effect.

[0064] The technical solutions of the embodiments of this application are described in detail below with reference to the accompanying drawings.

[0065] Figure 1 This is a flowchart illustrating the entity recognition method based on the attention-based Bi-LSTM+CRF model as shown in the embodiments of this application.

[0066] See Figure 1 An entity recognition method based on an attention-based Bi-LSTM+CRF model includes:

[0067] In step S101, the training set is preprocessed to obtain the input vector of the training set. The input vector includes character granularity vector, word granularity vector, and dictionary granularity vector.

[0068] In one implementation, an annotation tool can be used to correctly annotate the entities in the text. Multiple texts with correctly annotated entities are combined into a training set. The texts in the training set are then transformed into vectors. The input vectors can include character-level vectors, word-level vectors, and dictionary-level vectors.

[0069] In step S102, the input vector is input into the attention-based Bi-LSTM+CRF model to be trained.

[0070] In step S103, in the input feature layer of the attention-based Bi-LSTM+CRF model to be trained, an attention mechanism is used to assign weights to the character-granular vectors, word-granular vectors, and vocabulary-granular vectors at different positions of the input vector, respectively, to obtain a weighted and mixed vector.

[0071] In one implementation, an attention mechanism can be introduced into the input feature layer of the attention-based Bi-LSTM+CRF model to be trained. Weights are assigned to character-level vectors, word-level vectors, and vocabulary-level vectors at different positions in the input vector. The sum of the weights of the character-level vectors, word-level vectors, and vocabulary-level vectors at the same position in the input vector is equal to 1. Based on the weights of the different granularity vectors at different positions in the input vector, a weighted sum operation is performed on the different granularity vectors at different positions in the input vector to obtain a mixed vector of the different granularity vectors at different positions in the mixed input vector. This mixed vector is the output of the input feature layer using the attention mechanism based on the input vector.

[0072] In step S104, the entity recognition result output by the attention-based Bi-LSTM+CRF model to be trained is obtained based on the hybrid vector.

[0073] In one implementation, the entity recognition result output by the attention-based Bi-LSTM+CRF model to be trained can be obtained by using the attention layer, forward LSTM, backward LSTM, and CRF layer of the Bi-LSTM+CRF model to be trained, based on the mixed vector output by the input feature layer.

[0074] In step S105, the loss function value is obtained based on the entity recognition results and the training set.

[0075] In one implementation, the loss function value of the attention-based Bi-LSTM+CRF model to be trained is calculated based on the entity recognition results output by the model and the entity annotations in the training set.

[0076] In step S106, if the loss function value is less than the set loss threshold and / or the number of iterative training iterations reaches the set number of iterations, the training of the attention-based Bi-LSTM+CRF model is considered complete.

[0077] In one implementation, if the loss function value is less than a set loss threshold, the training of the attention-based Bi-LSTM+CRF model is determined to be complete; and / or, if the number of iterative training iterations reaches a set number of iterations, the training of the attention-based Bi-LSTM+CRF model is determined to be complete.

[0078] In step S107, the text to be identified is preprocessed to obtain the input vector of the text to be identified.

[0079] In step S108, the input vector of the text to be identified is input into the trained attention-based Bi-LSTM+CRF model so that the attention-based Bi-LSTM+CRF model outputs the entity recognition result.

[0080] The entity recognition method based on the attention-based Bi-LSTM+CRF model illustrated in this application uses an input vector comprising three different granularities: character-level vectors, word-level vectors, and vocabulary-level vectors. In the input feature layer of the attention-based Bi-LSTM+CRF model, an attention mechanism is employed to assign weights to the three different granularities of the input vector at different positions, resulting in a weighted sum of the vectors. Based on this summed vector, the entity recognition result output by the attention-based Bi-LSTM+CRF model is obtained. This allows the attention-based Bi-LSTM+CRF model to dynamically assign different weights to the different granularities of the input vector during training, dynamically adjusting its fitting ability to the input vector. This comprehensively enhances the information contained in the input vector, enriches the information at each position within the vector, and enables better training of the attention-based Bi-LSTM+CRF model, improving training efficiency. It can accurately recognize entities in text, especially long texts, improving the accuracy and speed of entity recognition and achieving optimal entity recognition results.

[0081] Figure 2 This is another schematic diagram of the entity recognition method based on the attention-based Bi-LSTM+CRF model shown in the embodiments of this application. Figure 2 Compared to Figure 1 The scheme of this application is described in more detail.

[0082] See Figure 2 An entity recognition method based on an attention-based Bi-LSTM+CRF model includes:

[0083] In step S201, the text to be recognized is correctly labeled to obtain a training set.

[0084] In one implementation, an annotation tool can be used to correctly annotate entities in multiple long texts, and the correctly annotated long texts can be combined into a training set. For example, entities in multiple travelogues can be correctly annotated, and the training set can be composed of multiple correctly annotated travelogues.

[0085] In step S202, the text in the training set is subjected to vector transformations at three different granularities to obtain the input vector of the training set. The input vector includes character-granular vector, word-granular vector, and vocabulary-granular vector.

[0086] In one implementation, the text of the training set can be vectorized at the character level to obtain a character-level vector; at the word level to obtain a word-level vector; or at the lexicon level to obtain a lexicon-level vector. Lexicon granularity directly indicates whether the current word is in a pre-stored POI (Point of Interest) database, improving the ability to recognize existing POIs. The pre-stored POI database contains information on attractions, restaurants, hotels, etc. If the current word is in the pre-stored POI database, its vector is 1; if it is not, its vector is 0.

[0087] In one implementation, such as Figure 3 As shown, taking the text "Shuangyue Bay in Huizhou is beautiful" as an example, the text is segmented at the character granularity, word granularity, and lexicon granularity to obtain three different vectors corresponding to the three different granularities of the text: character granularity vector L1, word granularity vector L2, and lexicon granularity vector L3. The input vector L of the text includes character granularity vector L1, word granularity vector L2, and lexicon granularity vector L3.

[0088] In one implementation, the entire training text can be segmented using the jieba word segmentation tool, and word-granular vectors can be obtained through the word2vec (word2vector) word vector model. The segmented text can then be trained using the skipgram model of word2vec. The process can be iterated until the stopping condition is met, resulting in 128-dimensional word vectors for each word.

[0089] In one implementation, the length of the text can be represented by the number of characters or words, and the length of the text in the training set can be judged. If the text exceeds 200 characters or words, the text is truncated and divided into multiple input texts with a maximum length of 200 characters or words.

[0090] In one implementation, text truncation can be performed using common punctuation marks (e.g., commas, periods, question marks, exclamation marks) as delimiters, dividing the text into multiple input texts of up to 200 characters or words in length. If the character count exceeds 200 at a certain delimiter, the truncation position of the current input text can be moved to the previous delimiter. For example, in the sentence "Where to go for National Day ******* (first sentence), why not consider ********* (second sentence), Shanghai Putuoshan ****** (third sentence)," if the truncation position is at the period of the third sentence, and the length of the input text composed of the first, second, and third sentences exceeds 200 characters, then the truncation position is placed at the comma (,) of the second sentence, with the first and second sentences treated as one input text, and the third sentence treated as a separate input text.

[0091] In step S203, the input vector of the training set is input into the attention-based Bi-LSTM+CRF model to be trained. The attention-based Bi-LSTM+CRF model includes an input feature layer that introduces an attention mechanism, a first attention layer set before the forward LSTM, and a second attention layer set before the backward LSTM.

[0092] In one implementation, such as Figure 4 As shown, the attention-based Bi-LSTM+CRF model includes an input feature layer 401 that introduces an attention mechanism, a first attention layer 402 set before the forward LSTM 403, a second attention layer 404 set before the backward LSTM 405, and a CRF layer 406.

[0093] In step S204, in the input feature layer, an attention mechanism is used to assign weights to the character-level vector, word-level vector, and lexicon-level vector at each different position of the input vector. Based on the weights of the character-level vector, word-level vector, and lexicon-level vector at each different position of the input vector, the attention mechanism is used to perform a weighted sum operation on the character-level vector, word-level vector, and lexicon-level vector at each different position of the input vector to obtain a weighted sum operation mixed vector.

[0094] In one implementation, in the input feature layer that incorporates an attention mechanism, weights are assigned to the character-level vector, word-level vector, and lexicon-level vector at each different position of the input vector. The sum of the three weights for the same position in the input vector is equal to 1. Based on the weights of these weights, the attention mechanism is used to perform a weighted sum operation on each of the input vectors, resulting in a mixed vector of the character-level vector, word-level vector, and lexicon-level vector at each different position of the mixed input vector. This mixed vector is then output by the input feature layer based on the attention mechanism.

[0095] like Figure 3 As shown, the input vector for the text "Shuangyue Bay in Huizhou is beautiful" includes: a character-level vector, a word-level vector, and a lexicon-level vector. The weights of the character-level vector, word-level vector, and lexicon-level vector at the first position of the input vector are a, b, and c, respectively, where a + b + c = 1. The mixed vector at the first position = character-level vector * a + word-level vector * b + lexicon-level vector * c. The weights of the character-level vector, word-level vector, and lexicon-level vector at the second position of the input vector are a², b², and c², respectively, where a² + b² + c² = 1. The two-position blend vector = character granularity vector at the second position * a² + word granularity vector at the second position * b² + lexicon granularity vector at the second position * c²; ...; and so on. Weights are assigned to the character granularity vector, word granularity vector, and lexicon granularity vector at each different position of the input vector. A weighted sum is then performed on these vectors to obtain the blend vector of the blended character granularity vector, word granularity vector, and lexicon granularity vector at each different position of the input vector. This yields the blend vector output by the input feature layer based on the attention mechanism of the input vector.

[0096] In one implementation, the input vector of the training set consists of vectors at three different granularities: character-level vectors, word-level vectors, and dictionary-level vectors. Therefore, the weights of the character-level, word-level, and dictionary-level vectors at each different position in the input vector can form a 3x3 matrix, where each value represents the weight of the three different granularities at each position in the input vector. During model training, the model corrects the values ​​in the 3x3 matrix during backpropagation of the loss function, thereby learning the respective weights of the character-level, word-level, and dictionary-level vectors at each position in the input vector.

[0097] In step S205, the hybrid vector is input into the first attention layer, and the attention mechanism is used to obtain the first weighted sum vector output by the first attention layer.

[0098] In one implementation, the hybrid vector output from the input feature layer is input into the first Attention layer. In the first Attention layer, an attention mechanism is employed to obtain the similarity between pairwise positions in the hybrid vector. Based on the similarity between pairwise positions, different weights are assigned to different positions in the hybrid vector. Based on the different weights at different positions in the hybrid vector, a weighted sum operation is performed on the hybrid vector to obtain a first weighted sum vector with attention information, thus obtaining the first weighted sum vector output by the first Attention layer.

[0099] In step S206, the first weighted sum vector is input into the forward LSTM to obtain the forward fusion vector output by the forward LSTM.

[0100] In one implementation, the first weighted sum vector output by the first Attention layer can be input into the forward LSTM. The forward LSTM performs forward fusion on the information of the first weighted sum vector and outputs a forward fused vector to obtain the first weighted sum vector output by the forward LSTM.

[0101] In step S207, the forward fusion vector is input into the second attention layer, and the attention mechanism is used to obtain the second weighted sum vector.

[0102] In one implementation, the forward fusion vector output from the forward LSTM can be input into a second Attention layer. In the second Attention layer, an attention mechanism is used to assign different weights to different positions of the forward fusion vector; based on the different weights at different positions, a weighted sum operation is performed on the forward fusion vector to obtain a second weighted sum vector with attention information; the second weighted sum vector output by the second Attention layer is then obtained.

[0103] In step S208, the second weighted sum vector is input into the backward LSTM to obtain the backward fusion vector output by the backward LSTM.

[0104] In one implementation, the second weighted sum vector output from the second Attention layer can be input into the backward LSTM. The backward LSTM performs reverse fusion on the information of the second weighted sum vector and outputs a backward fused vector, thus obtaining the backward fused vector output by the backward LSTM.

[0105] In step S209, the backward fusion vector is input into the CRF layer to obtain the entity recognition result output by the CRF layer.

[0106] In one implementation, the backward fusion vector output from the backward LSTM can be input into the CRF layer. The CRF layer can use 's' to label independent characters, 'b' to label the start of an entity, 'm' to label the temporal region of an entity, and 'e' to label the end of an entity. Using the backward fusion vector as input, the CRF layer labels the vector at each position. Based on the label of each vector, the classification label result is obtained, and the entity recognition result of the training set is output. For example, the text to be recognized, "Huizhou's Shuangyue Bay is beautiful," has the label "besbmess." The CRF layer can label and classify which of the 'sbme' elements each position in the vector sequence belongs to. Based on the 'be' in the classification result, it can be determined which word is the entity; that is, the string between 'be' is an entity.

[0107] In step S210, the loss function value is calculated based on the entity recognition results and the training set.

[0108] In step S211, it is determined whether the loss function value is less than the set loss threshold or whether the number of iterative training iterations has reached the set number of iterations; if yes, step S212 is executed; if no, step S204 is executed.

[0109] In one implementation, the loss function value and / or the number of iterations of the attention-based Bi-LSTM+CRF model can be judged. If the loss function value is less than a set loss threshold and / or the number of iterations reaches a set number of iterations, step S212 is executed; if the loss function value is greater than or equal to the set loss threshold or the number of iterations does not reach the set number of iterations, step S204 is executed to continue iterative training of the attention-based Bi-LSTM+CRF model until the loss function value of the attention-based Bi-LSTM+CRF model is less than the set loss threshold and / or the number of iterations reaches the set number of iterations.

[0110] In step S212, it is determined that the training of the attention-based Bi-LSTM+CRF model is complete.

[0111] In one implementation, if the loss function value is less than a set loss threshold, it can be determined that the training of the attention-based Bi-LSTM+CRF model is complete; and / or, if the number of iterative training iterations of the attention-based Bi-LSTM+CRF model reaches a set number of iterations, it can also be determined that the training of the attention-based Bi-LSTM+CRF model is complete.

[0112] It is understandable that the training of the attention-based Bi-LSTM+CRF model is considered complete when either the loss function value is less than the set loss threshold or the number of iterative training iterations reaches the set number of iterations; alternatively, the training of the attention-based Bi-LSTM+CRF model is considered complete when both conditions are met.

[0113] In step S213, the text to be identified is preprocessed to obtain the input vector of the text to be identified; the input vector of the text to be identified is then input into the trained attention-based Bi-LSTM+CRF model so that the attention-based Bi-LSTM+CRF model outputs the entity recognition result.

[0114] In one implementation, the text to be identified can be vectorized at the character level to obtain the character-level vector of the training set; at the word level to obtain the word-level vector of the training set; or at the lexicon level to obtain the lexicon-level vector of the training set. An input vector for the text to be identified is then obtained, comprising character-level, word-level, and lexicon-level vectors. This input vector is then fed into a trained attention-based Bi-LSTM+CRF model to obtain the labeled text and output the entity recognition result based on the labeled text.

[0115] In one implementation, the Viterbi algorithm can be used to decode the text's tags. The Viterbi algorithm works from the beginning, calculating the sequence probability of tags up to the current position based on the previous position. It then selects the sequence of tags with the highest probability at the last input position as the combined result sequence to obtain the text entity.

[0116] The entity recognition method based on the attention-based Bi-LSTM+CRF model illustrated in this application uses an input vector comprising three different granularities: character-level vectors, word-level vectors, and vocabulary-level vectors. In the input feature layer of the attention-based Bi-LSTM+CRF model, an attention mechanism is employed to assign weights to vectors of different granularities at different positions within the input vector, resulting in a weighted sum of the vectors. Based on this summed vector, the entity recognition result output by the attention-based Bi-LSTM+CRF model is obtained. This allows the attention-based Bi-LSTM+CRF model to dynamically assign different weights to vectors of different granularities at different positions within the input vector during training, dynamically adjusting its fitting ability to the input vector. This comprehensively enhances the information contained in the input vector, enriches the information at each position within the vector, and enables better training of the attention-based Bi-LSTM+CRF model, improving training efficiency. It can accurately recognize entities in text, especially long texts, improving the accuracy and speed of entity recognition and achieving optimal entity recognition results.

[0117] Furthermore, the entity recognition method based on the attention-based Bi-LSTM+CRF model shown in the embodiments of this application includes a first attention layer set before the forward LSTM and a second attention layer set before the backward LSTM. It can comprehensively and diversely obtain information in the vector, improve the accuracy of entity recognition of text, especially long text, meet the needs of various application scenarios, and achieve the best entity recognition effect.

[0118] Corresponding to the aforementioned application function implementation method embodiments, this application also provides an entity recognition device, electronic device, and corresponding embodiments based on an attention-based Bi-LSTM+CRF model.

[0119] Figure 5 This is a schematic diagram of the structure of an entity recognition device based on an attention-based Bi-LSTM+CRF model, as shown in an embodiment of this application.

[0120] See Figure 5 An entity recognition device based on an attention-based Bi-LSTM+CRF model includes a preprocessing module 501, a vector input module 502, a vector mixing module 503, a result acquisition module 504, a loss calculation module 505, and a training completion module 506.

[0121] The preprocessing module 501 is used to preprocess the training set to obtain the input vector of the training set. The input vector includes character granular vector, word granular vector, and dictionary granular vector.

[0122] In one implementation, an annotation tool can be used to correctly annotate the entities in the text, and multiple texts with correctly annotated entities can be combined into a training set. The preprocessing module 501 performs vector transformation on the text in the training set, and the input vectors may include character-level vectors, word-level vectors, and dictionary-level vectors.

[0123] The vector input module 502 is used to input the input vector obtained by the preprocessing module 501 into the attention-based Bi-LSTM+CRF model to be trained.

[0124] The vector mixing module 503 is used to apply an attention mechanism to the input feature layer of the attention-based Bi-LSTM+CRF model to be trained, and to assign weights to the character-granular vector, word-granular vector, and vocabulary-granular vector at different positions of the input vector input by the vector input module 502, respectively, to obtain a weighted and mixed vector.

[0125] In one implementation, the vector mixing module 503 can introduce an attention mechanism into the input feature layer of the attention-based Bi-LSTM+CRF model to be trained. Weights are assigned to the character-level vectors, word-level vectors, and vocabulary-level vectors at different positions in the input vector input to the vector input module 502, respectively. The sum of the weights of the character-level vectors, word-level vectors, and vocabulary-level vectors at the same position in the input vector is equal to 1. Based on the weights of the different granularity vectors at different positions in the input vector, a weighted sum operation is performed on the different granularity vectors at different positions in the input vector to obtain a mixed vector of the different granularity vectors at different positions in the mixed input vector. This results in a mixed vector output by the input feature layer using the attention mechanism based on the input vector.

[0126] The result acquisition module 504 is used to obtain the entity recognition result output by the attention-based Bi-LSTM+CRF model to be trained based on the mixed vector obtained by the vector mixing module 503.

[0127] In one implementation, the result acquisition module 504 can obtain the entity recognition result output by the attention-based Bi-LSTM+CRF model to be trained, based on the mixed vector obtained by the vector mixing module 503, through the attention layer, forward LSTM, backward LSTM, and CRF layer of the attention-based Bi-LSTM+CRF model to be trained.

[0128] The loss calculation module 505 is used to obtain the loss function value based on the entity recognition results and training set obtained by the result acquisition module 504.

[0129] In one implementation, the loss calculation module 505 calculates the loss function value of the attention-based Bi-LSTM+CRF model to be trained based on the entity recognition results output by the Bi-LSTM+CRF model to be trained and the entity annotations in the training set.

[0130] The training completion module 506 is used to determine that the training of the attention-based Bi-LSTM+CRF model is complete if the loss function value obtained by the loss calculation module 505 is less than the set loss threshold and / or the number of iterative training cycles reaches the set number of iterations.

[0131] In one implementation, if the loss function value obtained by the loss calculation module 505 is less than a set loss threshold, the training completion module 506 determines that the training of the attention-based Bi-LSTM+CRF model is complete; and / or, when the number of iterative training iterations reaches a set number of iterations, the training completion module 506 determines that the training of the attention-based Bi-LSTM+CRF model is complete.

[0132] The preprocessing module 501 is also used to preprocess the text to be recognized to obtain the input vector of the text to be recognized.

[0133] The vector input module is also used to input the input vector of the text to be recognized obtained by the preprocessing module 501 into the attention-based Bi-LSTM+CRF model after training, so that the attention-based Bi-LSTM+CRF model outputs the entity recognition result.

[0134] The technical solution illustrated in this application's embodiments includes three different granularities of input vectors: character-level vectors, word-level vectors, and dictionary-level vectors. In the input feature layer of the attention-based Bi-LSTM+CRF model, an attention mechanism is employed to assign weights to vectors of different granularities at different positions within the input vector, resulting in a weighted sum of the resulting mixed vectors. Based on this mixed vector, the entity recognition result output by the attention-based Bi-LSTM+CRF model is obtained. This allows the attention-based Bi-LSTM+CRF model to dynamically assign different weights to vectors of different granularities at different positions within the input vector during training, dynamically adjusting its fitting ability to the input vector. This comprehensively enhances the information contained in the input vector, enriches the information at each position within the vector, and enables better training of the attention-based Bi-LSTM+CRF model, improving training efficiency. It also enables accurate entity recognition in text, especially long texts, improving the accuracy and speed of entity recognition and achieving optimal entity recognition results.

[0135] Figure 6This is another schematic diagram of the entity recognition device based on the attention-based Bi-LSTM+CRF model shown in the embodiments of this application.

[0136] See Figure 6 An entity recognition device based on an attention-based Bi-LSTM+CRF model includes a preprocessing module 501, a vector input module 502, a vector mixing module 503, a result acquisition module 504, a loss calculation module 505, a training completion module 506, and a judgment module 601.

[0137] The function of the loss calculation module 505 can be found in [reference]. Figure 5 As shown.

[0138] The preprocessing module 501 is used to perform vector transformations on the text of the training set at three different granularities to obtain the input vector of the training set. The input vector includes character-granular vector, word-granular vector, and dictionary-granular vector.

[0139] In one implementation, the preprocessing module 501 can perform vector transformation on the training set text at the character granularity to obtain the character-granularity vector of the training set; it can perform vector transformation on the training set text at the word granularity to obtain the word-granularity vector of the training set; and it can perform vector transformation on the training set text at the lexicon granularity to obtain the lexicon granularity vector of the training set. Lexicon granularity directly indicates whether the current word is in a pre-stored POI (Point of Interest) database, which can improve the recognition ability of existing POIs. The pre-stored POI database stores information on POIs such as scenic spots, restaurants, and hotels. If the current word is in the pre-stored POI database, the preprocessing module 501 sets the vector of the current word to 1; if the current word is not in the pre-stored POI database, the preprocessing module 501 sets the vector of the current word to 0.

[0140] The vector input module 502 is used to input the input vector of the training set obtained by the preprocessing module 501 into the attention-based Bi-LSTM+CRF model to be trained. The attention-based Bi-LSTM+CRF model includes an input feature layer that introduces an attention mechanism, a first attention layer set before the forward LSTM, and a second attention layer set before the backward LSTM.

[0141] The vector mixing module 503 includes a weight assignment submodule 5031 and a vector operation submodule 5032.

[0142] The weight assignment submodule 5031 is used to assign weights to the character-granular vector, word-granular vector, and vocabulary-granular vector at each different position of the input vector input by the vector input module 502 in the input feature layer of the attention-based Bi-LSTM+CRF model to be trained, using an attention mechanism.

[0143] The vector operation submodule 5032 is used to assign weights to the character-granular vector, word-granular vector, and lexicon-granular vector at each different position of the input vector given by the submodule 5031 according to the weights. It adopts an attention mechanism to perform weighted sum operations on the character-granular vector, word-granular vector, and lexicon-granular vector at each different position of the input vector input by the vector input module 502, and obtains the weighted sum operation mixed vector.

[0144] The result acquisition module 504 is further configured to input the mixed vector obtained by the vector mixing module 503 into the first attention layer of the attention-based Bi-LSTM+CRF model to be trained, and use the attention mechanism to obtain the first weighted sum vector output by the first attention layer; input the first weighted sum vector into the forward LSTM of the attention-based Bi-LSTM+CRF model to be trained, and obtain the forward fusion vector output by the forward LSTM; input the forward fusion vector into the second attention layer of the attention-based Bi-LSTM+CRF model to be trained, and use the attention mechanism to obtain the second weighted sum vector; input the second weighted sum vector into the backward LSTM of the attention-based Bi-LSTM+CRF model to be trained, and obtain the backward fusion vector output by the backward LSTM; and input the backward fusion vector into the CRF layer of the attention-based Bi-LSTM+CRF model to be trained, and obtain the entity recognition result output by the CRF layer.

[0145] The judgment module 601 is used to determine whether the loss function value obtained by the loss calculation module 505 is less than the set loss threshold or whether the number of iterative training cycles has reached the set number of iterations.

[0146] In one implementation, if the judgment module 601 determines that the loss function value is greater than or equal to the set loss threshold and / or the number of iterative training iterations has not reached the set number of iterations, the vector mixing module 503 can use an attention mechanism in the input feature layer of the attention-based Bi-LSTM+CRF model to be trained to assign weights to vectors of different granularities at each different position of the input vector input by the vector input module 502, and obtain a weighted mixed vector.

[0147] In one implementation, the judgment module 601 can judge the loss function value and / or the number of iterative training cycles of the attention-based Bi-LSTM+CRF model obtained by the loss calculation module 505. If the judgment module 601 judges that the loss function value is less than a set loss threshold and / or the number of iterative training cycles reaches a set number of iterations, the training completion module 506 determines that the training of the attention-based Bi-LSTM+CRF model is complete. If the judgment module 601 judges that the loss function value is greater than or equal to the set loss threshold or the number of iterative training cycles does not reach the set number of iterations, the vector mixing module 503, the result acquisition module 504, the loss calculation module 505, and the judgment module 601 continue to perform iterative training on the attention-based Bi-LSTM+CRF model until the judgment module 601 judges that the loss function value of the attention-based Bi-LSTM+CRF model is less than the set loss threshold and / or the number of iterative training cycles reaches a set number of iterations, and the training completion module 506 determines that the training of the attention-based Bi-LSTM+CRF model is complete.

[0148] The preprocessing module 501 is also used to preprocess the text to be recognized to obtain the input vector of the text to be recognized.

[0149] The vector input module 502 is also used to input the input vector of the text to be recognized obtained by the preprocessing module 501 into the attention-based Bi-LSTM+CRF model after training, so that the attention-based Bi-LSTM+CRF model outputs the entity recognition result.

[0150] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated further here.

[0151] Figure 7 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application.

[0152] See Figure 7 The electronic device 70 includes a memory 701 and a processor 702.

[0153] The processor 702 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0154] Memory 701 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by processor 702 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation. Furthermore, memory 701 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (e.g., DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, memory 701 may include a removable storage device that is readable and / or writable, such as a laser disc (CD), a read-only digital multifunction optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, a high-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.

[0155] The memory 701 stores executable code, which, when processed by the processor 702, can cause the processor 702 to execute part or all of the methods described above.

[0156] Furthermore, the method according to this application can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing some or all of the steps in the method described above.

[0157] Alternatively, this application may be implemented as a computer-readable storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium) storing executable code (or computer program or computer instruction code) thereon, which, when executed by a processor of an electronic device (or server, etc.), causes the processor to perform part or all of the steps of the methods described above according to this application.

[0158] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An entity recognition method based on an attention-based Bi-LSTM+CRF model, characterized in that, include: The training set is preprocessed to obtain the input vector of the training set. The input vector includes a character granularity vector, a word granularity vector, and a lexicon granularity vector, wherein the lexicon granularity vector is a vector indicating whether the words in the text of the training set are in a pre-stored interest point library. The input vector is input into the attention-based Bi-LSTM+CRF model to be trained, wherein the attention-based Bi-LSTM+CRF model includes an input feature layer, a first attention layer set before the forward LSTM, and a second attention layer set before the backward LSTM. In the input feature layer of the attention-based Bi-LSTM+CRF model to be trained, an attention mechanism is used to assign weights to the character granular vector, word granular vector, and vocabulary granular vector at different positions of the input vector, respectively, to obtain a weighted sum of the mixed vectors. Based on the hybrid vector, the entity recognition result output by the attention-based Bi-LSTM+CRF model to be trained is obtained, including: inputting the hybrid vector into the first attention layer, using an attention mechanism to obtain a first weighted sum vector output by the first attention layer, inputting the first weighted sum vector into the forward LSTM to obtain a forward fusion vector output by the forward LSTM, inputting the forward fusion vector into the second attention layer, using an attention mechanism to obtain a second weighted sum vector, inputting the second weighted sum vector into the backward LSTM to obtain a backward fusion vector output by the backward LSTM, and inputting the backward fusion vector into the CRF layer to obtain the entity recognition result output by the CRF layer; Based on the entity recognition results and the training set, the loss function value is obtained; If the loss function value is less than the set loss threshold and / or the number of iterative training iterations reaches the set number of iterations, the training of the attention-based Bi-LSTM+CRF model is considered complete. The text to be identified is preprocessed to obtain the input vector of the text to be identified; The input vector of the text to be identified is input into the trained attention-based Bi-LSTM+CRF model so that the attention-based Bi-LSTM+CRF model outputs the entity recognition result.

2. The method according to claim 1, characterized in that, In the input feature layer of the attention-based Bi-LSTM+CRF model to be trained, an attention mechanism is used to assign weights to the character-level vector, word-level vector, and vocabulary-level vector at different positions of the input vector, respectively, to obtain a weighted sum of the resulting mixed vector, including: In the input feature layer of the attention-based Bi-LSTM+CRF model to be trained, an attention mechanism is used to assign weights to the character granular vector, word granular vector, and vocabulary granular vector at each different position of the input vector. Based on the weights of the character-level vector, word-level vector, and dictionary-level vector at each different position in the input vector, an attention mechanism is used to perform a weighted sum operation on the character-level vector, word-level vector, and dictionary-level vector at each different position in the input vector, resulting in a weighted sum hybrid vector.

3. The method according to claim 1, characterized in that, The method further includes: If the loss function value is greater than or equal to the set loss threshold and / or the number of iterations for training does not reach the set number of iterations, the following step is executed: in the input feature layer of the attention-based Bi-LSTM+CRF model to be trained, an attention mechanism is used to assign weights to the character granular vector, word granular vector, and dictionary granular vector at different positions of the input vector, respectively, to obtain a weighted sum of the mixed vectors.

4. An entity recognition device based on an attention-based Bi-LSTM+CRF model, characterized in that, include: The preprocessing module is used to preprocess the training set to obtain the input vector of the training set. The input vector includes a character granularity vector, a word granularity vector, and a lexicon granularity vector, wherein the lexicon granularity vector is a vector indicating whether the words in the text of the training set are in a pre-stored interest point library. The vector input module is used to input the input vector obtained by the preprocessing module into the attention-based Bi-LSTM+CRF model to be trained, wherein the attention-based Bi-LSTM+CRF model includes an input feature layer, a first attention layer set before the forward LSTM, and a second attention layer set before the backward LSTM. The vector mixing module is used to assign weights to the character-granular vector, word-granular vector, and vocabulary-granular vector at different positions of the input vector input by the vector input module in the input feature layer of the attention-based Bi-LSTM+CRF model to be trained, using an attention mechanism, to obtain a weighted and mixed vector. The result acquisition module is used to obtain the entity recognition result output by the attention-based Bi-LSTM+CRF model to be trained based on the mixture vector obtained by the vector mixing module. This includes: inputting the mixture vector obtained by the vector mixing module into the first attention layer of the attention-based Bi-LSTM+CRF model to be trained; using an attention mechanism to obtain a first weighted sum vector output by the first attention layer; inputting the first weighted sum vector into the forward LSTM of the attention-based Bi-LSTM+CRF model to be trained; obtaining a forward fusion vector output by the forward LSTM; inputting the forward fusion vector into the second attention layer of the attention-based Bi-LSTM+CRF model to be trained; using an attention mechanism to obtain a second weighted sum vector; inputting the second weighted sum vector into the backward LSTM of the attention-based Bi-LSTM+CRF model to be trained; obtaining a backward fusion vector output by the backward LSTM; and inputting the backward fusion vector into the CRF layer of the attention-based Bi-LSTM+CRF model to be trained; and obtaining the entity recognition result output by the CRF layer. The loss calculation module is used to obtain the loss function value based on the entity recognition result obtained by the result acquisition module and the training set; The training completion module is used to determine that the training of the attention-based Bi-LSTM+CRF model is complete if the loss function value obtained by the loss calculation module is less than a set loss threshold and / or the number of iterative training cycles reaches a set number of iterations. The preprocessing module is also used to preprocess the text to be identified, and obtain the input vector of the text to be identified. The vector input module is further configured to input the input vector of the text to be identified obtained by the preprocessing module into the attention-based Bi-LSTM+CRF model after training, so that the attention-based Bi-LSTM+CRF model outputs the entity recognition result.

5. The apparatus according to claim 4, characterized in that, The vector mixing module includes: The weight assignment submodule is used to assign weights to the character-granular vector, word-granular vector, and vocabulary-granular vector at each different position of the input vector input by the vector input module in the input feature layer of the attention-based Bi-LSTM+CRF model to be trained, using an attention mechanism. The vector operation submodule is used to assign weights to the character-granular vector, word-granular vector, and lexicon-granular vector at each different position of the input vector given by the submodule according to the weights. Using an attention mechanism, the submodule performs a weighted sum operation on the character-granular vector, word-granular vector, and lexicon-granular vector at each different position of the input vector input by the vector input module to obtain a weighted sum operation hybrid vector.

6. An electronic device, characterized in that, include: processor; as well as A memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 1-3.

7. A computer-readable storage medium, characterized in that: It stores executable code that, when executed by a processor of an electronic device, causes the processor to perform the method as described in any one of claims 1-3.

Images