A nested entity recognition method, device and storage medium integrating full-text information

Abstract

The present invention discloses a nested entity recognition method, device and storage medium that integrates full-text information. The method comprises the following steps: using a pre-trained language model to encode the text to be recognized, obtaining H L and H L‑1 ; Classify and identify whether all words in the text are the beginning and end boundaries of entities; match all the identified beginning and end boundaries one by one to obtain multiple candidate entities; L‑1 The concatenation mapping of the first and last word vectors at the corresponding positions in H is used as the candidate entity vector, and each candidate entity vector is regarded as the query vector. L‑1 Treating the data as a sequence of keyword vectors and a sequence of value vectors, a relative position-based attention mechanism and residual connections are used to obtain candidate entity vectors that incorporate the full-text information. The candidate entity vectors that incorporate the full-text information are then used as features to determine the candidate entity type. The method of the present invention features a simple sequence and a rational design, low computational complexity, and is applicable to any form of entity nesting.

Description

Technical Field

[0001] The present invention relates to the technical field of information extraction of machine learning and natural language processing, and relates to a nested entity recognition method, device and storage medium integrated with full-text information. Background Art

[0002] The task of entity recognition is to automatically identify and classify named entities from a given unstructured text. Common entity types include place names and personal names, but there is no single definition; definitions vary depending on the specific task. Today, with the highly developed internet, massive amounts of unstructured text are generated every minute. Accurately identifying meaningful entities within this text will undoubtedly improve the performance of other related natural language processing tasks, such as information search, query understanding, relationship extraction, and intelligent question-answering. With the widespread application of entity recognition technology, the previously overlooked problem of nested entities has gradually attracted the attention of researchers. Nested entities are common in unstructured text in some fields, but traditional entity recognition research has not specifically designed to address them, resulting in a decrease in entity recognition accuracy.

[0003] Traditional entity recognition typically uses sequence tagging to categorize and parse each character into entities. However, for nested entities, each character may belong to multiple entities, making traditional sequence tagging methods inadequate. This issue directly impacts the effectiveness of applications like recommendation and search. Therefore, efficient tagging and decoding of nested entities in text will be a key research focus for nested entity recognition. Summary of the Invention

[0004] Since the prior art has the above-mentioned defects, the present invention provides a two-stage nested entity recognition method that incorporates full-text information, which overcomes the defect that the prior art sequence labeling method is not applicable to nested entities.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A nested entity recognition method incorporating full-text information includes the following steps:

[0007] 1) Use the pre-trained language model to encode the text to be recognized and obtain the top-level output vector sequence H of the model L and the next top-level output vector sequence H L-1 ;

[0008] 2) Classify and identify whether all characters in the text are the beginning and end boundaries of entities;

[0009] 3) Pair all identified head and tail boundaries one by one to obtain multiple candidate entities;

[0010] 4) Place the candidate entity in H L-1 The concatenation of the first and last word vectors at the corresponding position in H is used as the candidate entity vector, and each candidate entity vector is regarded as the query vector. L-1 Treated as a sequence of keyword vectors and a sequence of value vectors, the full-text information vector of each candidate entity is obtained based on the attention mechanism of relative position, and the candidate entity vector that integrates the full-text information is obtained through residual connection;

[0011] 5) Determine the candidate entity type using the candidate entity vector that integrates the full-text information as a feature.

[0012] The nested entity recognition method that incorporates full-text information of the present invention has a simple step sequence and a reasonable design, low computational complexity, and avoids traversing all entity possibilities through a two-stage approach; the method is applicable to any form of entity nesting and still has a good effect on non-nested entity recognition, has good applicability, and has great application prospects.

[0013] As the preferred technical solution:

[0014] As described above, a nested entity recognition method that incorporates full-text information, wherein the pre-trained language model is a BERT model, is only provided as a feasible technical solution, and other pre-trained language models may also be applicable;

[0015] The step 2) is specifically as follows:

[0016] H L The vector corresponding to each word in is used to make two binary classification predictions respectively to determine whether it is the first character of the entity and whether it is the last character of the entity.

[0017] In the above-mentioned nested entity recognition method integrating full-text information, the step 3) is specifically as follows:

[0018] According to the first and last boundary judgment results obtained in step 2), all identified first boundaries are paired with the following last boundaries including the current position to obtain candidate entities.

[0019] In the above-mentioned nested entity recognition method that incorporates full-text information, the generation process of the candidate entity vector is specifically as follows:

[0020] Take the first and last positions of the candidate entities in H L-1 The corresponding vectors in the vector sequence are concatenated and linearly transformed as the vector representation of the candidate entity:

[0021]

[0022] where Q ij Represents the vector representation of the candidate entity starting at position i and ending at position j, concat is the concatenation function, and Represents the vector of the i-th, j-th position output by the L-1 layer of the pre-trained language model, W ENT 、b ENT are learnable parameters.

[0023] In the above-mentioned nested entity recognition method that incorporates full-text information, the calculation formula for relative position embedding in the relative position-based attention mechanism is as follows:

[0024] In this relative position-based attention mechanism, each word or character will have a head and tail position index. The relative position of any candidate entity and the character in the original text can be accurately described by the following four formulas:

[0025]

[0026]

[0027]

[0028]

[0029] where x a 、x b Represents any candidate entity, start and end represent the first and last positions of the entity, respectively. Indicates the relative distance between the first characters of two words. The same is true for x b is the word in the original text. The first and last positions of the words in the original text are the same, so and equal, and are equal, so we can only take and Characterize candidate entity x a and the word x in the original text b The relative position is then embedded by index and mapped to obtain the vector representation of the relative position:

[0030]

[0031]

[0032]

[0033] Among them, R ab Represents the candidate entity x a and the word x in the original text b The vector representation of the relative position, W Ris a learnable parameter, P is the relative position embedding, d is the relative position, the generation rule of P is consistent with that used in the original Transformer, and the odd-even interleaving uses sin and cos to generate fixed position embedding, and dim is the dimension of the position embedding.

[0034] In the above-mentioned nested entity recognition method that incorporates full-text information, the calculation formula of the attention in the relative position-based attention mechanism is as follows:

[0035]

[0036] Among them, A is the attention score matrix, A ab Represents the candidate entity x a In the original text, word x b The attention score, and Represent the candidate entity x a and the bth word x in the original text b The vector representation of W q 、W k,E 、W k,R , u and v are learnable parameters. Combined with the multi-head attention mechanism in Transformer, the relative position attention calculation in this chapter can be described as:

[0037] head c =Attn c (Q,H L-1 )

[0038] MultiHead(Q,H L-1 )=concat(head1,head2,…,head h )W MH

[0039]

[0040] Among them head c is the output of the c-th head, Q is the entity vector sequence corresponding to the candidate entity set, W MH is a learnable parameter, h is the total number of heads, and z is a scaling constant.

[0041] In the above-mentioned nested entity recognition method integrating full-text information, the calculation formula for obtaining the candidate entity vector integrating full-text information through residual connection is as follows:

[0042] score=softmax(concat(MultiHead(Q,H L-1 ),Q)W O )

[0043] Where score is the normalized score of the entity type, W O It is a trainable parameter, and the argmax function takes the index with the largest type score.

[0044] As described above, a nested entity recognition method that integrates full-text information, wherein the candidate entity type is judged based on the candidate entity vector that integrates full-text information as a feature, refers to considering the type corresponding to the maximum score as the type of each candidate entity, wherein the special type NONE is set to indicate that the candidate entity is an unreasonable entity.

[0045] The present invention further provides a computer device, comprising:

[0046] at least one processor; and,

[0047] a memory communicatively connected to the at least one processor; wherein,

[0048] The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the nested entity recognition method integrated with full-text information as described above is implemented.

[0049] In addition, the present invention also provides a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the nested entity recognition method integrated with full-text information as described above is implemented.

[0050] The above technical solution is only a feasible technical solution of the present invention. The protection scope of the present invention is not limited thereto. Those skilled in the art can reasonably adjust the specific design according to actual needs.

[0051] The above invention has the following advantages or beneficial effects:

[0052] (1) The nested entity recognition method of the present invention, which integrates full-text information, has a simple sequence and a reasonable design, low computational complexity, and avoids traversing all entity possibilities through a two-stage approach;

[0053] (2) The nested entity recognition method of the present invention that integrates full-text information is applicable to any form of entity nesting and still has a good effect on non-nested entity recognition. It has good applicability and great application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The present invention and its features, configurations, and advantages will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings. Like reference numerals indicate like parts throughout the drawings. The drawings are not drawn to scale, emphasis instead being placed on illustrating the subject matter of the present invention.

[0055] Figure 1 This is an overall flow chart of the nested entity recognition method that incorporates full-text information of the present invention;

[0056] Figure 2 is a schematic diagram of candidate entity boundary pairing;

[0057] Figure 3 is a schematic structural diagram of a computer device of the present invention;

[0058] Among them, 1-computer equipment, 11-processor, 12-memory. DETAILED DESCRIPTION

[0059] The structure of the present invention is further described below with reference to the accompanying drawings and specific embodiments, but is not intended to limit the present invention.

[0060] Example 1

[0061] A nested entity recognition method that incorporates full-text information, the steps are as follows Figure 1 As shown, it specifically includes the following steps:

[0062] S1: Use the pre-trained language model to encode the text to be recognized to obtain the text vector sequence H L and H L-1 ;

[0063] S2: Classify and identify whether all characters in the text are the beginning and end boundaries of entities;

[0064] S3: Pair all identified head and tail boundaries one by one to obtain multiple candidate entities;

[0065] S4: Input the position of each candidate entity and the text vector sequence H L-1 , the candidate entity vector integrating the full-text information is obtained through relative position-based attention mechanism calculation and residual connection;

[0066] S5: Determine the candidate entity type based on the candidate entity vector that integrates the full-text information. The special type NONE is set to indicate that the candidate entity is an unreasonable entity.

[0067] Step S1 is specifically as follows:

[0068] The text to be recognized is converted into a token sequence through a predefined vocabulary, and the token sequence is input into a pre-trained language model (such as BERT). The current pre-trained language model is composed of a multi-layer network stack. The top-level output H of the pre-trained language model is taken. L and the second top layer H L-1 is the vector sequence of the text to be recognized.

[0069] Step S2 is specifically as follows:

[0070] The vector sequence H L Input to the linear classification layer and use the sigmoid function to normalize the scores of the start and end boundaries to 0 to 1, setting all outputs greater than the set threshold to 1:

[0071] B s =ε(sigmoid(H L W S +b S )-t)

[0072] B e =ε(sigmoid(H L W E +b E )-t)

[0073] Among them B s and B e Indicates the start boundary and end boundary marking results, ε is a switching function, the input is greater than zero and outputs 1, otherwise it outputs 0, H L is the top-level output vector sequence of the BERT model, W S 、W E 、b S and b E It is a learnable parameter for determining the start and end boundaries, and t is the set threshold.

[0074] Step S3 is specifically as follows:

[0075] Will start boundary marking result B s All the words corresponding to the output 1 in the result are regarded as the start boundary of the entity, and the end boundary is marked as the result B e All the words corresponding to the output of 1 in are regarded as the end boundary of the entity, and the start boundary is paired with the end boundary of the current position and all subsequent positions to obtain all possible candidate entities, such as Figure 2 shown.

[0076] Step S4 is specifically as follows:

[0077] The candidate entity is placed in H L-1 The first and last word vectors of the corresponding positions in the text are concatenated and mapped as candidate entity vectors, and each candidate entity vector is regarded as a query vector. The text vector sequence H L-1 It is regarded as a sequence of keyword vectors and a sequence of value vectors. The full-text information vector of each candidate entity is obtained based on the attention mechanism of relative position, and the candidate entity vector that integrates the full-text information is obtained through residual connection.

[0078] First, generate the candidate entity vector, take the first and last positions of the candidate entity in H L-1 The corresponding vectors in the vector sequence are concatenated and linearly transformed as the vector representation of the candidate entity:

[0079]

[0080] where Q ij Represents the vector representation of the candidate entity starting at position i and ending at position j, concat is the concatenation function, and Represents the vector of the i-th and j-th positions output by the L-1 layer of the BERT model, W ENT 、b ENT are learnable parameters.

[0081] Then, the relative position embedding of the candidate entity and each word in the original text is calculated. In this relative position-based attention mechanism method, each word or character will have a head and tail position index. The relative position of any candidate entity and the word in the original text can be accurately described by the following four formulas:

[0082]

[0083]

[0084]

[0085]

[0086] where x a 、x b Represents any candidate entity, start and end represent the first and last positions of the entity, respectively. Indicates the relative distance between the first characters of two words. The same is true for x b is the word in the original text. The first and last positions of the words in the original text are the same, so and equal, and are equal, so we can only take and Characterize candidate entity x a and the word x in the original text b The relative position is then embedded by index and mapped to obtain the vector representation of the relative position:

[0087]

[0088]

[0089]

[0090] Among them, R ab Represents the candidate entity x a and the word x in the original text bThe vector representation of the relative position, W R is a learnable parameter, P is the relative position embedding, d is the relative position, the generation rule of P is consistent with that used in the original Transformer, and the odd-even interleaving uses sin and cos to generate fixed position embedding, and dim is the dimension of the position embedding.

[0091] Then the attention calculation based on relative position is performed. The attention calculation in the attention mechanism based on relative position is as follows:

[0092]

[0093] Among them, A is the attention score matrix, A ab Represents the candidate entity x a In the original text, word x b The attention score, and Represent the candidate entity x a and the bth word x in the original text b The vector representation of W q 、W k,E 、W k,R , u and v are learnable parameters. Combined with the multi-head attention mechanism in Transformer, the relative position attention calculation in this chapter can be described as:

[0094] head c =Attn c (Q,H L-1 )

[0095] MultiHead(Q,H L-1 )=concat(head1,head2,…,head h )W MH

[0096]

[0097] Among them head c is the output of the c-th head, Q is the entity vector sequence corresponding to the candidate entity set, W MH is a learnable parameter, h is the total number of heads, and z is a scaling constant.

[0098] Finally, the vector obtained by the attention calculation based on the relative position is regarded as the full-text information vector and the original candidate entity vector is concatenated and mapped to obtain the candidate entity vector that integrates the full-text information. The normalized score of the entity type is obtained through softmax:

[0099] score=softmax(concat(MultiHead(Q,H L-1),Q)W O )

[0100] Among them, score is the normalized score of the entity type, W O For trainable parameters, the argmax function takes the index step with the largest type score.

[0101] S5 is specifically:

[0102] Analyze the type of each candidate entity. There will be a maximum value in the normalized score corresponding to each candidate entity. The index corresponding to this value is regarded as the type of the candidate entity, and each index value is defined as an entity type. The type of non-valid entities is defined as NONE. Candidate entities classified as NONE are excluded from the final recognition result.

[0103] At this point, the nested entity recognition method that integrates full-text information has been completed.

[0104] The nested entity recognition method that integrates full-text information provided by the embodiment of the present invention abandons the traditional sequence labeling method and uses a two-stage entity recognition method that combines boundary recognition with type judgment. This method can theoretically mark any nested entity form.

[0105] Example 2

[0106] A computer device, the computer device 1 is as follows Figure 3 As shown, it includes: at least one processor 11; and a memory 12 that is communicatively connected to the at least one processor 11; wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, it implements the nested entity recognition method that integrates full-text information as described in Example 1.

[0107] Example 3

[0108] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the nested entity recognition method integrated with full-text information as described in Example 1.

[0109] Those skilled in the art should understand that they can implement variations by combining the prior art with the above embodiments, which will not be described in detail here. Such variations do not affect the essence of the present invention and will not be described in detail here.

[0110] The above describes the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the above-mentioned specific embodiments, and the devices and structures that are not described in detail should be understood to be implemented in a common manner in the art; any technician familiar with the art can use the above-mentioned disclosed methods and technical contents to make many possible changes and modifications to the technical solutions of the present invention without departing from the scope of the technical solutions of the present invention, or modify them into equivalent embodiments of equivalent changes, which does not affect the essential content of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention that do not depart from the content of the technical solutions of the present invention are still within the scope of protection of the technical solutions of the present invention.

Claims

1. A nested entity recognition method that incorporates full-text information, characterized in that: The following steps are involved: 1) Use the pre-trained language model to encode the text to be recognized and obtain the top-level output vector sequence of the model and the next top-level output vector sequence ; 2) Classify and identify whether all characters in the text are the beginning and end boundaries of entities; 3) Pair all identified head and tail boundaries one by one to obtain multiple candidate entities; 4) Place the candidate entity in The concatenation of the first and last word vectors at the corresponding positions in the is used as the candidate entity vector, and each candidate entity vector is regarded as the query vector. Treated as a sequence of keyword vectors and a sequence of value vectors, the full-text information vector of each candidate entity is obtained based on the attention mechanism of relative position, and the candidate entity vector that integrates the full-text information is obtained through residual connection; The calculation formula for relative position embedding in the relative position-based attention mechanism is as follows: ， ， ， ， ， in, is a candidate entity, express and The relative distance to the first position, Represents candidate entities and the original words The vector representation of the relative position of is a learnable parameter, P is the relative position embedding, The generation rule of P is consistent with that used in the original Transformer paper. Sin and cos are used alternately to generate fixed position embeddings. Dim is the dimension of the position embedding. The calculation formula of the attention in the relative position-based attention mechanism is as follows: ， in, Represents candidate entities In the original text, word b The attention score, and Represent candidate entities and the bth word in the original text The vector representation of v is a learnable parameter; Combined with the multi-head attention mechanism in Transformer, the relative position attention calculation can be described as: ， ， ， in is the entity vector sequence corresponding to the candidate entity set, is a learnable parameter, h is the total number of heads, and z is a scaling constant; 5) Determine the candidate entity type using the candidate entity vector that integrates the full-text information as a feature.

2. The nested entity recognition method according to claim 1, wherein: The pre-trained language model is a BERT model; The step 2) is specifically as follows: Will The vector corresponding to each word in is used to make two binary classification predictions respectively to determine whether it is the first character of the entity and whether it is the last character of the entity.

3. The method for identifying nested entities incorporating full-text information according to claim 1, wherein: The step 3) is specifically as follows: According to the first and last boundary judgment results obtained in step 2), all identified first boundaries are paired with the following last boundaries including the current position to obtain candidate entities.

4. The method for identifying nested entities incorporating full-text information according to claim 1, wherein: The generation process of the candidate entity vector is specifically as follows: Take the first and last positions of the candidate entity The corresponding vectors in the vector sequence are concatenated and linearly transformed as the vector representation of the candidate entity: ， in Indicates Position start, The vector representation of the candidate entity at the end of the position, is the splicing function, and Represents the output of the L-1 layer of the pre-trained language model The vector of position, are learnable parameters.

5. The method for identifying nested entities incorporating full-text information according to claim 1, wherein: The calculation formula for the candidate entity vector obtained by residual connection and fused with full-text information is as follows: ， in is the normalized score of the entity type, is a trainable parameter.

6. The method for identifying nested entities incorporating full-text information according to claim 5, characterized in that: The method of determining the candidate entity type by taking the candidate entity vector integrating the full-text information as a feature refers to considering the type corresponding to the maximum score as the type of each candidate entity, wherein setting the special type NONE indicates that the candidate entity is an unreasonable entity.

7. A computer device, characterized in that: The computer device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the nested entity recognition method integrated with full-text information as described in any one of claims 1 to 6 is implemented.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the nested entity recognition method integrated with full-text information as described in any one of claims 1 to 6 is implemented.

Images