A non-continuous entity recognition method based on sequence to forest

Abstract

The present invention proposes a non-continuous entity recognition method based on sequence to forest. In order to better characterize the dependencies between internal blocks of entities, a forest structure is used to model entity sets in texts. A neural network-based "encoder-decoder" generation framework is adopted to realize the sequence to forest generation process. On the encoder side, a Transformer-based encoder is first used to capture the global dependency features of each word in the text, and then a convolutional neural network is used to further capture the local dependency features of the words. On the decoder side, an attention module based on a "note" mechanism is designed to capture the semantic relevance between the output and input at each decoding moment. The beneficial effects of the present invention are: it can ensure the disorder between entities and the validity within the entity, it can effectively capture the relevance between entity blocks, it can effectively improve the model's recognition ability for non-continuous entities, and it can be adapted to the scenario of continuous entity recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a sequence-to-forest based discontinuous entity recognition method. Background Art

[0002] Entity recognition, a fundamental task in natural language processing and a core technology in information extraction, aims to identify entity boundaries and classify entities from free text. However, most current entity recognition methods assume that entities are continuous, meaning they consist of consecutive words in the text. They are unable to identify non-continuous entities, meaning they consist of discontinuous words in the text. For example, the text sequence "Tsinghua University, Peking University Library" contains two discontinuous entities, "Tsinghua University Library" and "Peking University Library."

[0003] Currently, mainstream non-continuous entity recognition methods can be divided into two categories: sequence labeling-based methods and sequence generation-based methods. Sequence labeling-based methods primarily expand BIO tags into more complex tags to identify the common parts and non-continuous parts of entities; sequence generation-based methods primarily transform extractive tasks into generative tasks, that is, they implement the generation process of text sequences into entity sequences based on sequence-to-sequence generative models. Although the above two methods can solve the problem of non-continuous entity recognition to a certain extent, they both have their own shortcomings: sequence labeling-based methods are limited by limited labels and cannot mark entity boundaries, which is prone to structural ambiguity and seriously affects entity recognition performance; although sequence generation-based methods overcome structural ambiguity, they model the entity set in the text as a linear sequence structure, which introduces unnecessary entity order prediction, further increasing the difficulty of the model, making the model overly dependent on the predefined entity order in the training set and prone to false negative predictions.

[0004] Based on the above background, the focus and difficulty of non-continuous entity recognition lies in determining multiple entity blocks (or spans) belonging to the same entity and capturing the correlation between entity blocks (or spans), while ensuring the disorder between entities. Summary of the Invention

[0005] In order to address the above-mentioned deficiencies in the prior art, the present invention provides a sequence-to-forest based non-continuous entity recognition method. By introducing the entity set in the forest structure modeling text, the internal composition of non-continuous entities and the dependencies between them can be clearly characterized; based on the "encoder-decoder" generation framework, the generation process of modeling sequence (input text) to forest (output entity) can ensure the disorder between entities and the order within the entity; at the same time, by designing an attention module at the decoding end, the correlation between entity blocks and the semantic correlation between output and input can be effectively captured, thereby effectively improving the decoding accuracy.

[0006] The purpose of the present invention is achieved through the following technical solutions.

[0007] A non-continuous entity recognition method based on sequence to forest includes the following steps:

[0008] Step 1: For the input text, use the Transformer encoder to obtain the context encoding vector of each word in the text. At this time, the word vector has the global dependency characteristics of the word; at the same time, the sentence encoding vector can be obtained.

[0009] Step 2: For the word vector sequence with global word dependency features, a convolutional neural network is applied to further capture the local word dependency features to obtain a word vector sequence that combines global and local dependency features.

[0010] Step 3: For the sentence encoding vector, a decoding skeleton based on an LSTM neural network is used to capture the dependencies between time steps in the decoding time series and obtain the hidden vector at each decoding moment.

[0011] In step 4, for the word vector sequence that integrates global and local dependency features and the hidden vector at each decoding moment, an attention module based on the "note" mechanism is used to capture the semantic association between the output and input at each decoding moment and update the encoded word vector sequence.

[0012] Step 5: For the updated encoded word vector sequence, use the multi-label classification module to perform forest structure decoding in the order of "entity type-entity block 1-entity block 2-entity block 3" to predict the result of each decoding moment.

[0013] Step 6: Perform joint inference on the prediction results at each decoding moment, i.e., entity type, entity block 1, entity block 2, and entity block 3, to parse out the entities in the text.

[0014] Furthermore, the method as a whole adopts an "encoder-decoder" generation framework, wherein steps 1-2 are executed on the encoder side, and steps 3-6 are executed on the decoder side.

[0015] Furthermore, in step 1, the following steps are also performed:

[0016] Step 1.1: For each word in the input text, obtain its multidimensional word vector sequence, which mainly consists of three parts: fixed pre-trained encoding representation, character-level encoding representation, and positional encoding representation.

[0017] In step 1.2, the multidimensional word vector sequence is input into the Transformer encoder. By stacking L layers of Transformer backbone modules (Transformer Block), the global dependency features of each word are learned to obtain the context-aware word vector sequence and sentence encoding vector. The specific formula is as follows:

[0018]

[0019] in, represents the output of the Transformer encoder, x i ,…,x N Represents the multidimensional word vector representation of each word in the input text, and N represents the length of the text sequence. The sentence encoding vector of the input sequence is the output at the last moment, that is,

[0020] Furthermore, in step 2, a convolutional neural network is used to further capture the local dependency features of words. The specific formula is as follows:

[0021]

[0022] Among them, vector c=[c1,…,c N ] represents a word vector sequence that combines the global and local dependency features of words, which can provide expressive encoding representation for the decoding end.

[0023] Furthermore, in step 3, a decoding skeleton based on an LSTM neural network is used to capture the dependencies between time steps in the time series, specifically:

[0024] Given the input of the current time step is represented as e t , the hidden vector of the previous time step is represented as s t-1 , the hidden vector representation s at the current time step t The specific calculation formula is as follows:

[0025] s t =LSTM(e t ,s t-1 ) (3)

[0026] in, In order to enable batch calculation during training, the number of entity blocks contained in all entities is set to 3, that is, the decoding length is set to 4 (1 entity block + 3 entity types), that is, t = ∈ {1, 2, 3, 4}; the decoding order is set to: "entity type-entity block 1-entity block 2-entity block 3".

[0027] Furthermore, in step 4, an attention module based on a "notebook" mechanism is used to capture the semantic association between the output and input at each decoding moment and update the encoded word vector sequence, including:

[0028] First, according to the hidden vector s at each moment t The encoding representation of the previous moment Perform attention calculation to learn which information in the input sequence the prediction information at the current moment will focus on. The specific calculation formula is as follows:

[0029]

[0030] Then, the convolutional neural network is used to further update the encoding vector representation. The specific calculation formula is as follows:

[0031]

[0032] in, That is, the word vector sequence obtained at the encoding end integrates global and local dependencies, and [;] represents the splicing operation.

[0033] Furthermore, in step 5, a multi-label classification module is used to perform forest structure decoding in the order of "entity type-entity block 1-entity block 2-entity block 3", including:

[0034] When predicting the entity type (i.e., t=1), we first perform a maximum pooling operation on the updated encoding vector sequence, and then predict the entity type in the text. The specific formula is as follows:

[0035]

[0036] Among them, σ(·) represents the sigmoid function, Max(·) represents the maximum pooling operation, W and b represent learnable parameters, and p type Represents the predicted probability of the entity type.

[0037] When predicting entity blocks (i.e., t = 2, 3, 4), we first use two different multi-layer perceptrons (MLPs) to map the updated encoded representation sequence into the entity’s start and end word spaces, and then use bilinear attention calculations. The specific formula is as follows:

[0038]

[0039]

[0040]

[0041] Where σ(·) represents the sigmoid function, Indicates that the i-th word in the sequence is the starting word of the entity at the t-th decoding time, Indicates that the jth word in the sequence is the end word of the entity at the tth decoding time, U represents the learnable parameter, and p(i,j) represents the current predicted entity block as span (w i ,w j ), (w i ,w j ) represents the continuous segment (or span) from the i-th word to the j-th word in the input text.

[0042] Furthermore, in step 6, the entities in the text are parsed by combining the prediction results at each decoding moment, including:

[0043] During the training phase, the best prediction probability of all decoding moments is estimated, and the optimization goal is to minimize the negative log-likelihood function. The specific calculation formula is as follows:

[0044]

[0045] Among them, X is the input text, θ is the model parameter, span1 * ,span2 * ,type * Represents the true value.

[0046] During the testing phase, based on the prediction results at each moment, for each branch of each tree in the forest, the entity is spliced ​​according to "entity block 1-entity block 2-entity block 3" to obtain the entity, and finally the (entity, entity type) pair is obtained.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] 1. By using a forest structure to model the entity set in the text, the present invention can more clearly express the internal composition of the entity and the dependency relationship between entity blocks compared to the linear sequence structure, and has a shorter decoding path length. It can not only ensure the disorder between entities and the order within the entity, but also improve the decoding speed.

[0049] 2. The present invention integrates the global and local dependency features of words at the decoding end. The global dependency features of words can be captured through the Transformer encoder. Furthermore, a convolutional neural network is applied to the word vector sequence with global dependency features to capture the local dependency of words, which is conducive to providing expressive word encoding vector representation for entity recognition.

[0050] 3. At the decoding end, the present invention designs an attention module based on the "note" mechanism, which can capture the semantic correlation between the output and input at each decoding moment, and can further capture the dependency between entity types and entity blocks, and between entity blocks, greatly improving the performance of non-continuous entity recognition.

[0051] 4. The present invention uses a generative paradigm to model entity recognition tasks, which has strong generalization. It not only has good non-continuous entity recognition performance, but is also adaptable to continuous entity recognition scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A workflow diagram of non-continuous entity recognition based on sequence to forest provided by an embodiment of the present invention;

[0053] Figure 2 A schematic diagram of an example of using a forest structure to represent entity sets in text provided by an embodiment of the present invention;

[0054] Figure 3 A model structure diagram of a sequence-to-forest based non-continuous entity recognition method provided by an embodiment of the present invention;

[0055] Figure 4 A structural diagram of the attention module based on the "sticky note" mechanism provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The present invention is described in detail below with reference to the accompanying drawings: This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process. It should be pointed out that any variation and replacement based on the concept of the present invention falls within the scope of protection of the present invention.

[0057] Existing methods for non-contiguous entity recognition based on sequence annotation are limited by a limited set of labels and cannot annotate entity boundaries. This leads to severe structural ambiguity, which reduces the performance of non-contiguous entity recognition. Although methods based on sequence-to-sequence generation can avoid structural ambiguity, they force the prediction of entity order, which increases the difficulty of the model and causes the model to be overly dependent on the entity order in the training set, which can easily lead to overfitting and false negative predictions.

[0058] The present invention proposes a sequence-to-forest based non-continuous entity recognition method. This method can clearly characterize the internal composition of non-continuous entities and the dependencies between them by introducing the entity set in the forest structure modeling text; based on the "encoder-decoder" generation framework, the generation process of modeling sequence (input text) to forest (output entity) can ensure the disorder between entities and the order within the entities; at the same time, by designing an attention module at the decoding end, the correlation between entity blocks and the semantic correlation between output and input can be effectively captured, thereby effectively improving the decoding accuracy.

[0059] like Figure 1 As shown, in this embodiment, a non-continuous entity recognition method based on sequence to forest adopts an end-to-end generation paradigm as a whole, including two parts: encoding and decoding. The specific execution steps are as follows:

[0060] Step 1: For the input text, use the Transformer encoder to obtain the context encoding vector of each word in the text. At this time, the word vector has the global dependency characteristics of the word; at the same time, the sentence encoding vector can be obtained.

[0061] Step 2: For the word vector sequence with global word dependency features, a convolutional neural network is applied to further capture the local word dependency features to obtain a word vector sequence that combines global and local dependency features.

[0062] Step 3: For the sentence encoding vector, a decoding skeleton based on an LSTM neural network is used to capture the dependencies between time steps in the decoding time series and obtain the hidden vector at each decoding moment.

[0063] In step 4, for the word vector sequence that integrates global and local dependency features and the hidden vector at each decoding moment, an attention module based on the "note" mechanism is used to capture the semantic association between the output and input at each decoding moment and update the encoded word vector sequence.

[0064] Step 5: For the updated encoded word vector sequence, use the multi-label classification module to perform forest structure decoding in the order of "entity type-entity block 1-entity block 2-entity block 3" to predict the result of each decoding moment.

[0065] Step 6: Perform joint inference on the prediction results at each decoding moment, i.e., entity type, entity block 1, entity block 2, and entity block 3, to parse out the entities in the text.

[0066] like Figure 2As shown in the upper part, in this embodiment, the example sentence using the forest structure to represent the entity set in the text contains two entities in the entity set: {"Couldn't walk comfortably", "Couldn't sleep comfortably"}, both of which are non-continuous entities and the entity type is ADR (adverse drug reaction).

[0067] The present invention uses a forest structure to represent entity sets in sentences: entity sets of different entity types constitute different trees in the forest, that is, the root node of each tree represents the entity type, and the remaining nodes represent the entity blocks that constitute the entity. A path from the root node to the leaf node represents an entity belonging to the current entity type. Figure 2 As shown in the lower part: the forest contains only one tree, indicating that the sentence contains only one entity type; the tree has two branches, indicating that there are two entities belonging to this entity type in the sentence; for example, the leftmost branch represents the entity of type ADR (adverse drug reaction): "Couldn't sleep comfortably", which consists of two entity blocks "(0, 2)" and "(6, 7)" in sequence.

[0068] like Figure 3 As shown, in this embodiment, a non-continuous entity recognition method based on sequence to forest adopts an "encoder-decoder" model structure as a whole. Specifically, the encoder side mainly performs feature encoding representation for the input text, which mainly includes two parts: Transformer encoder and convolutional neural network encoder; the decoder side mainly decodes the entity set of the forest structure based on the encoding representation, and each decoding unit mainly includes three parts: a decoding skeleton based on an LSTM neural network, an attention module based on a "note" mechanism, and a multi-label classification module.

[0069] Furthermore, before inputting into the Transformer encoder, we first obtain the multidimensional word vector representation x for each word in the input text. i , mainly consists of three parts: fixed pre-trained encoding representation Character-level encoding representation and positional encoding representation The specific formula is as follows:

[0070]

[0071] Where N represents the length of the sentence sequence, and the [;] operation represents concatenation.

[0072] Furthermore, for the multidimensional vector representation x of the word obtained i, is input into the Transformer encoder. By stacking L layers of Transformer backbone modules (Transformer Block), the global dependency information and deep semantic information of each word are learned to obtain a context-aware word vector sequence. The specific formula is as follows:

[0073]

[0074] in, Represents the output of the Transformer encoder. The sentence of the input sequence represents the output of the vector at the last moment, that is,

[0075] Furthermore, for the context-aware word vector sequence, a convolutional neural network is used to further capture the local dependency features of words. The specific formula is as follows:

[0076]

[0077] The vector representation c at this time incorporates the global dependency and local dependency features of the word, providing the decoder with an expressive encoding representation.

[0078] At this point, the encoding operation has been completed, and the sentence encoding representation and the word vector sequence that integrates global and local dependency features have been obtained. The next step is to perform the decoding operation. In order to achieve parallel training of the model, the span number (or entity block number) of all entities is set to 3, that is, the decoding length is set to 4 (1 entity type + 3 entity blocks), that is, t = ∈ {1, 2, 3, 4}; if the entity blocks contained in an entity are less than 3, then "(-1, -1)" is used for padding; set the starting identifier of the model decoding end to " <bos>"; Set the decoding order to: "entity type-entity block 1-entity block 2-entity block 3".

[0079] Furthermore, for the obtained sentence encoding representation, the decoding skeleton based on LSTM neural network can effectively capture the dependency between time steps in the time series. Given the input representation of the current time step as e t , the hidden vector of the previous time step is represented as s t-1 , the hidden vector representation s at the current time step t It can be calculated by the following formula:

[0080] s t =LSTM(e t ,s t-1 ) (4)

[0081] in, Depending on the time step t, e t There are three main types of forecast information:

[0082] (1) When t=1, the encoding of initialization e1 is expressed as Because the first starting input of the decoder is always a special character " <bos>", the current prediction is entity type: type k .

[0083] (2) When t = 2, the input vector e2 is the vector representation corresponding to the entity type output in the previous time step The prediction at the current moment is the first entity block (entity block 1) that constitutes the entity:

[0084] (3) When t=3 or t=4, the input vector e3 / e4 ​​is the entity block output in the previous time step The corresponding vector representation The prediction at the current moment is the second entity block or the third entity block (entity block 2 / entity block 3) of the entity:

[0085] Among them, the entity block s b:e Vector representation of It contains three parts of information: entity boundary, entity length, and entity internal words. The specific formula is as follows:

[0086]

[0087]

[0088]

[0089]

[0090] t∈{3,4} (9)

[0091] Furthermore, for the hidden vector obtained at the decoding moment and the word vector sequence output by the encoding end that combines global and local information, the attention module based on the "sticky note" mechanism regards the encoding vector representation as a "convenient note" to record and track the information of each decoding moment.

[0092] like Figure 4 As shown, in this embodiment, a structural process of an attention module based on the "note" mechanism includes: first, according to the hidden vector s at each moment t The encoding representation of the previous moment Perform attention calculations to learn which information in the input sequence the current prediction information will focus on, and then use the convolutional neural network to further update the encoded representation. The specific calculation formula is as follows:

[0093]

[0094]

[0095] in, That is, the word encoding sequence obtained at the encoding end integrates global and local dependencies.

[0096] Furthermore, for the updated encoding representation The multi-label classification module is used to treat the prediction at each moment as a multi-label classification task. As mentioned above, for each entity, the prediction order is: "entity type - entity block 1 - entity block 2 - entity block 3". Based on the decoding order, there are two main prediction situations:

[0097] When predicting the entity type (i.e., t=1), we first perform a max-pooling operation on the updated encoded representation sequence, and then predict the entity type in the text. The specific formula is as follows:

[0098]

[0099] Among them, σ(·) represents the sigmoid function, Max(·) represents the maximum pooling operation, W and b represent learnable parameters, and p ty0e Represents the predicted probability of the entity type.

[0100] like Figure 3 At the decoding end in , the prediction result at the first decoding moment is entity type: ADR.

[0101] When predicting entity blocks (i.e., t = 2, 3, 4), we first use two different multi-layer perceptrons to map the updated encoded representation sequence into the entity's start word space and end word space, and then use bilinear attention calculation. The specific calculation formula is as follows:

[0102]

[0103]

[0104]

[0105] Where σ(·) represents the sigmoid function, Indicates that the i-th word in the sequence is the starting word of the entity at the t-th decoding time, Indicates that the jth word in the sequence is the end word of the entity at the tth decoding time, U represents the learnable parameter, and p(i,j) represents the current predicted entity block as span (w i ,w j ), (w i ,w j ) represents the continuous segment (or span) from the i-th word to the j-th word in the input text.

[0106] like Figure 3 At the decoding end in , the prediction results for the second decoding moment are: "(0, 2)" and "(0, 3)", indicating that there are two entities of type ADR in the sentence and the representations of their first entity blocks are "(0, 2)" (the span from the 0th word to the 2nd word in the sentence) and "(0, 3)" (the span from the 0th word to the 3rd word in the sentence). Similarly, the prediction result for the third decoding moment can be obtained at the decoding end. Through the above operations, it can be seen that nodes at the same layer in each tree are decoded in parallel, and nodes at different layers are decoded sequentially, which can ensure disorder between entities and order within entities.

[0107] Furthermore, by combining the prediction results at each decoding moment, we can parse out the entities contained in the sentence. The operations in the training and testing phases are as follows:

[0108] During the training phase, the best prediction probability of all decoding moments is estimated, and the optimization goal is to minimize the negative log-likelihood function. The specific calculation formula is as follows:

[0109]

[0110]

[0111] Among them, X is the input text, θ is the model parameter, span1 * ,span2 * ,type * Represents the true value.

[0112] During the test phase, based on the prediction results at each moment, for each branch of each tree in the forest, the entity is spliced ​​according to "entity block 1-entity block 2-entity block 3" to obtain the entity, and finally the (entity, entity type) pair is obtained.

[0113] like Figure 3 At the decoding end in the first decoding moment, only one entity type, ADR, is obtained, indicating that the sentence contains only entities of the entity type ADR. The prediction results for "entity block 1-entity 2-entity block 3" on the two decoding paths are "(0, 3)-(7, 7)-(-1, -1)" and "(0, 2)-(6, 7)-(-1, -1)", respectively. By concatenating the spans (or entity blocks) they represent, two entities are obtained: "Couldn't walk comfortably" and "Couldn't sleep comfortably". Finally, combined with the entity types, the entity set contained in the sentence is: {(Couldn'twalk comfortably, ADR), (Couldn't sleep comfortably, ADR)}.

[0114] In order to verify the performance of the sequence-to-forest based non-continuous entity recognition method proposed in this invention in identifying non-continuous entities, the recognition verification of non-continuous entities was carried out based on two datasets containing non-continuous entities, ShARe13 and ShARe14. The ShARe13 and ShARe14 test sets were screened to construct a test dataset containing at least one non-continuous entity in the sentence, which were denoted as ShARe13-DS and ShARe14-DS respectively; in addition, a test dataset containing only non-continuous entities was constructed, which were denoted as ShARe13-DE and ShARe14-DE respectively. The experimental results are shown in Tables 1 and 2 when compared with the three currently better performing non-continuous entity recognition methods (Transition-based, Span-based and Seq2Seq-based).

[0115] From the given F1 value comparison results, it can be seen that the non-continuous entity recognition method of the present invention has significantly improved recognition performance on the two data sets compared with the sequence-to-sequence (Seq2Seq-based) method, thereby proving the effectiveness and feasibility of the method proposed in the present invention.

[0116] Table 1 Comparison of F1 performance based on ShARe13-DS and ShARe14-DS datasets

[0117] Model ShARe13-DS ShARe14-DS Transition-based 61.1 55.6 Span-based 56.3 59.9 Seq2Seq-based 64.8 65.7 Seq2Forest(ours) 66.0 67.1

[0118] Table 2 Comparison of F1 performance based on ShARe13-DE and ShARe14-DE datasets

[0119] Model ShARe13-DE ShARe14-DE Transition-based 49.0 43.8 Span-based 53.8 51.6 Seq2Seq-based 55.0 52.7 Seq2Forest(ours) 60.1 61.1

[0120] It is worth noting that the sequence-to-forest based non-continuous entity recognition method proposed in this invention can not only greatly improve the performance of non-continuous entity recognition, but also has good generalization. It can be applied to the scenario of continuous entity recognition tasks by simply adjusting the decoding step size.

[0121] For continuous entity recognition, we only need to set the length of each tree to 2, that is, set the decoding step size t∈{1,2}. Because continuous entities only contain one entity block, the decoder only needs to predict the entity type in the text at the first decoding time, and then predict the corresponding entity block for each entity type at the second decoding time.

[0122] The sequence-to-forest-based non-continuous entity recognition method provided by the above embodiment of the present invention uses a forest structure to model the entity set in the text. Compared with the linear sequence structure, it can more clearly express the internal composition of the entity and the dependency relationship between the blocks, has a shorter decoding path length, can ensure the disorder between entities and the order within the entity, and can also improve the decoding speed; at the decoding end, the global and local dependency features of the word are integrated, and the global dependency features of the word can be captured by the Transformer encoder. Furthermore, a convolutional neural network is applied to the word vector sequence with global dependency features to capture the local dependency of the word, which is conducive to providing a word vector representation with rich features for entity recognition; at the decoding end, an attention module based on the "note" mechanism is designed to capture the semantic correlation between the output and input at each decoding moment, and can further capture the dependency between the entity type and the entity block, and between the entity blocks, thereby greatly improving the performance of non-continuous entity recognition; using a generative paradigm to model the entity recognition task has strong generalization, not only has good non-continuous entity recognition performance, but also can be adapted to the scenario of continuous entity recognition.

[0123] The above is a detailed description of the specific embodiments of the present invention. It should be noted that the present invention is not limited to the implementation methods of the above specific embodiments. Any modifications and substitutions made by those skilled in the art based on the concept of the present invention fall within the scope of protection of the present invention.< / bos> < / bos>

Claims

1. A non-continuous entity recognition method based on sequence to forest, characterized by The steps are as follows: Step 1: For the input text, use the Transformer encoder to obtain the context encoding vector of each word in the text. At this time, the word vector has the global dependency characteristics of the word; at the same time, the sentence encoding vector can be obtained; Step 2: Apply a convolutional neural network to the word vector sequence with the global dependency features of the word to further capture the local dependency features of the word, and obtain a word vector sequence that combines the global and local dependency features; Step 3: For the sentence encoding vector, use a decoding skeleton based on an LSTM neural network to capture the dependencies between time steps in the decoding time series and obtain the hidden vector at each decoding moment; Step 4: For the word vector sequence that integrates global and local dependency features and the hidden vector at each decoding moment, an attention module based on the "note" mechanism is used to capture the semantic association between the output and input at each decoding moment and update the encoded word vector sequence, including: First, according to the hidden vector s at each moment t The encoding representation of the previous moment Perform attention calculation to learn which information in the input sequence the prediction information at the current moment will focus on. The specific calculation formula is as follows: Then, the convolutional neural network is used to further update the encoding vector representation. The specific calculation formula is as follows: in, That is, the word vector sequence obtained by the encoding end integrates global and local dependencies, and [;] represents the splicing operation; Step 5: For the updated encoded word vector sequence, use the multi-label classification module to perform forest structure decoding in the order of "entity type-entity block 1-entity block 2-entity block 3" to predict the result at each decoding moment; The multi-label classification module is used to perform forest structure decoding in the order of "entity type-entity block 1-entity block 2-entity block 3", including: When predicting the entity type, that is, t=1, we first perform a maximum pooling operation on the updated encoding vector sequence, and then predict the entity type in the text. The specific formula is as follows: Among them, σ(·) represents the sigmoid function, Max(·) represents the maximum pooling operation, W and b represent learnable parameters, and p type represents the predicted probability of the entity type; When predicting entity blocks, i.e., t=2, 3, and 4, we first use two different multi-layer perceptrons (MLPs) to map the updated encoded representation sequence into the entity’s start and end word spaces, and then use bilinear attention calculations. The specific formula is as follows: Where σ(·) represents the sigmoid function, Indicates that the i-th word in the sequence is the starting word of the entity at the t-th decoding time, Indicates that the jth word in the sequence is the end word of the entity at the tth decoding time, U represents the learnable parameter, and p(i,j) represents the current predicted entity block as span (w i ,w j ), (w i ,w j ) represents the continuous segment from word i to word j in the input text; Step 6: Perform joint inference on the prediction results at each decoding moment, i.e., entity type, entity block 1, entity block 2, and entity block 3, to parse out the entities in the text.

2. The method for non-continuous entity recognition based on sequence to forest according to claim 1, characterized in that: This method adopts an "encoder-decoder" generation framework, wherein steps 1-2 are executed on the encoder side, and steps 3-6 are executed on the decoder side.

3. The method for non-continuous entity recognition based on sequence to forest according to claim 1, characterized in that: In step 1, the global feature encoding of the input text includes: First, for each word in the input text, obtain its multidimensional word vector sequence, which mainly consists of three parts: fixed pre-trained encoding representation, character-level encoding representation, and positional encoding representation; Then, the multi-dimensional word vector sequence is input into the Transformer encoder. By stacking L layers of Transformer backbone modules, the global dependency features of each word are learned to obtain the context-aware word vector sequence and sentence encoding vector. The specific formula is as follows: in, represents the output of the Transformer encoder, x i ,…,x N Represents the multidimensional word vector representation of each word in the input text, N represents the length of the text sequence; x i ,…,x N The sentence encoding vector of the input sequence is the output at the last moment, that is, 4. The method for non-continuous entity recognition based on sequence to forest according to claim 1, characterized in that: In step 2, the application of convolutional neural network further captures the local dependency features of words. The specific formula is as follows: Among them, vector c=[c1,…,c N ] represents a word vector sequence that combines the global and local dependency features of words, which can provide expressive encoding representation for the decoding end.

5. The method for non-continuous entity recognition based on sequence to forest according to claim 1, characterized in that: In step 3, the decoding skeleton based on the LSTM neural network is used to capture the dependencies between time steps in the time series, specifically: Given the input of the current time step is represented as e t , the hidden vector of the previous time step is represented as s t-1 , the hidden vector representation s at the current time step t The specific calculation formula is as follows: s t =LSTM(e t ,s t-1 ) (3) Among them, In order to enable batch calculation during training, the number of entity blocks contained in all entities is set to 3, that is, the decoding length is set to 4, that is, "1 entity type + 3 entity blocks", that is, t = ∈ {1, 2, 3, 4}; the decoding order is set to: "entity type-entity block 1-entity block 2-entity block 3".

6. The method for non-continuous entity recognition based on sequence to forest according to claim 1, characterized in that: In step 6, the prediction results at each decoding moment are combined to parse out entities in the text, including: During the training phase, the best prediction probability of all decoding moments is estimated, and the optimization goal is to minimize the negative log-likelihood function. The specific calculation formula is as follows: Among them, X is the input text, θ is the model parameter, span1 * ,span2 * ,type * represents the true value; During the testing phase, based on the prediction results at each moment, for each branch of each tree in the forest, entities are spliced ​​according to "entity block 1-entity block 2-entity block 3" to obtain entities, and finally all entities in the text are obtained.

Images