A method for entity recognition and relation extraction that integrates lexical boundary and semantic information

Abstract

This invention relates to the field of natural language processing technology, and particularly to a method for entity recognition and relation extraction that integrates lexical boundaries and semantic information. The method includes constructing sample inputs and labels for a preprocessed language model; outputting the feature vector of the last layer through a BERT model; constructing task feature vectors for entity recognition and calculating the loss; constructing task feature vectors related to relation extraction and calculating the loss; adding the two losses by coefficients to obtain the total loss; and combining the results of entity recognition and relation extraction to obtain the final triplet. This invention addresses the problems of error accumulation, entity redundancy, and lack of interaction in deep learning methods; and solves the problem of extracting all possible fragment arrangements based on the existing method of displaying nested entities using fragment arrangement.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method for entity recognition and relation extraction that integrates lexical boundaries and semantic information. Background Technology

[0002] Entity recognition and relation extraction are important tasks in natural language processing, responsible for identifying entities from natural language text and extracting semantic relationships between entities.

[0003] Entity recognition and relation extraction based on deep learning pipeline methods refer to first identifying multiple different entities in a sentence, and then combining the identified entities to determine the relation type. The two processes are completely separated. Such methods suffer from problems such as error accumulation, entity redundancy, and missing interactions. However, joint extraction can effectively alleviate these problems.

[0004] For nested entities, based on the fragment arrangement method, all possible fragment arrangements are explicitly extracted. Since each selected fragment is independent, fragment-level features can be directly extracted to solve the problem. Summary of the Invention

[0005] To address the shortcomings of existing algorithms, this invention considers the constraints between the outputs of entity recognition models and relation extraction models. It employs a joint extraction method to identify semantic relationships between entities and entity pairs. Furthermore, it incorporates lexical boundaries and related semantic information, fully utilizing the positional relationships between different words in a sentence. Simultaneously, by combining sample sentences into predicted entity fragments before inputting them into a pre-trained language model, multiple predicted entity fragments can be identified simultaneously, improving model computational efficiency. Additionally, the fragment classification approach effectively addresses entity nesting. By utilizing entity recognition and relation extraction methods, this invention can efficiently identify entities and their types, accurately revealing the semantic relationships between entities, providing effective assistance for constructing knowledge graphs, intelligent question-answering systems, and more.

[0006] The technical solution adopted in this invention is: a method for entity recognition and relation extraction that integrates lexical boundaries and semantic information, comprising the following steps:

[0007] Step 1: Construct sample inputs and labels for the preprocessed language model;

[0008] Furthermore, specifically including:

[0009] Step 11: Segment the text statement into words, and add the [CLS] symbol to the segmented text statement to obtain the sequence {[CLS], T1, T2, T3, T...} i ..., T n}; where T iThis is represented as a token obtained after word segmentation of a text statement.

[0010] Step 12: Combine the m segments to be predicted from the end of the text statement, represented as {[CLS]; T1, T2, T3, ..., T...} n ;S1, S1,...,S1;S1,S2,S3,...,S m}; where {S1, S1, ..., S1} represents the first position information of the added segment to be predicted, {S1, S2, S3, ..., S1} represents the first position information of the added segment to be predicted. m} represents the tail position information of the segment to be predicted, until all segment position information S1 to S2 is traversed. n The position information of the added segment to be predicted shares the position information with the corresponding word unit in the text, resulting in a total of Z spliced ​​segments to be predicted;

[0011] Furthermore, the formula for calculating the Z spliced ​​segments to be predicted is as follows:

[0012]

[0013] Where L represents the length of the segment to be predicted, and N represents the number of tokens in the segmented text.

[0014] Step 13: Construct entity labels and relationship labels. Entity labels consist of entity boundary information and entity type label information. Relationship labels consist of the boundary information of the subject-object entity pair and the relationship type label.

[0015] Step 2: Input the samples into the pre-trained BERT model, and the BERT model outputs the feature vector of the last layer;

[0016] Furthermore, the feature vectors include word vectors and sentence vectors.

[0017] Step 3: Construct the task feature vector for entity recognition. The feature vector is obtained by concatenating the features after max pooling of the sentence, the predicted segment boundary word features, and the boundary features concatenated at the end of the sample. The data is fed into a NER classifier to obtain the classification result and the loss is calculated.

[0018] Furthermore, specifically including:

[0019] Step 31: Perform max pooling on the word vector feature information to obtain h. m The calculation formula is:

[0020] h m = maxpooling(h1,h2,...,h n );

[0021] Step 32: Concatenate partial feature vectors to obtain task features. The calculation formula is:

[0022]

[0023] Where cat represents the concatenate operation, h i h represents the first position information feature of the predicted entity segment. j This represents the tail position information feature of the predicted entity segment. This represents the first position information feature of the segment to be predicted added at the end of the sample. The tail position information features of the segment to be predicted are added to the end of the sample, and m candidate segments are identified by combining them at once;

[0024] Step 33, The data is fed into the NER classifier to obtain the prediction result for entity type k. The formula is:

[0025]

[0026] Among them, W e b e Let ε represent the trainable parameters of the task model for entity extraction and relation extraction, k represent the type of entity, and ε represent the set of entity types.

[0027] Step 34: Calculate the cross-entropy loss of the NER part. The formula is:

[0028]

[0029] Where N represents the number of samples, y ij Indicates whether it belongs to the current category.

[0030] Step 4: Construct task feature vectors related to the relation extraction task. The data is obtained by concatenating the sentence vector, the boundary features of the subject segment to be predicted, and the boundary features of the object segment to be predicted. This data is then fed into the RE classifier to obtain the classification result and calculate the loss.

[0031] Furthermore, specifically including:

[0032] Step 41: Concatenate partial feature vectors to obtain task features. The calculation formula is

[0033]

[0034] Where cat represents the concatenate operation, and h0 represents the [CLS] sentence feature vector output by the pre-trained language model; This is represented by the first position feature of the main segment a. This is represented by the tail position feature of the main segment a. This is represented by the first position feature of the candidate object segment. This is represented as the tail position feature of the candidate object segment;

[0035] Step S42, will The data is fed into the RE classifier, and the predicted relationship type between subject fragment a and object fragment b is 'l'. The formula is:

[0036]

[0037] Among them, W r b r Let represent the trainable parameters of the model, and l represent the type of relation. Represented as a set of relation types;

[0038] Step 43: Calculate the cross-entropy loss of the RE component. The formula is:

[0039]

[0040] Where N represents the number of samples, y ab Indicates whether it belongs to the current category.

[0041] Step 5: Determine the losses With loss The total loss is obtained by adding the coefficients together.

[0042] Furthermore, total losses The formula is:

[0043]

[0044] Where α and β are dynamic weights.

[0045] Step 6: Combine the results of entity recognition and relation extraction to obtain the final triples.<s,r,o> .

[0046] The beneficial effects of this invention are:

[0047] 1. Integrating lexical boundary information into sentences before inputting them into a pre-trained language model, and utilizing the positional relationships between words, can effectively improve model performance;

[0048] 2. Simultaneously calculating multiple segments to be predicted effectively improves the efficiency of model calculation;

[0049] 3. Employing a fragment-based classification method effectively solves the problem of entity overlap;

[0050] 4. The task model shares word vectors provided by the pre-trained language model BERT and incorporates relevant semantic information into the task feature vectors, enabling joint extraction to simultaneously complete entity recognition and relation extraction tasks. Attached Figure Description

[0051] Figure 1 This is a flowchart of the entity recognition and relation extraction method that integrates lexical boundaries and semantic information according to the present invention.

[0052] Figure 2 This is a schematic diagram of the input example of the preprocessed language model of the present invention;

[0053] Figure 3 This is a schematic diagram illustrating the process of constructing the feature vector for the entity recognition task of the present invention;

[0054] Figure 4 This is a schematic diagram illustrating the process of constructing feature vectors for the relation extraction task of this invention. Detailed Implementation

[0055] The present invention will be further described below with reference to the accompanying drawings and embodiments. The drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0056] like Figure 1 As shown, a method for entity recognition and relation extraction that integrates lexical boundaries and semantic information includes the following steps:

[0057] Step 1: Construct sample inputs and labels for the preprocessed language model;

[0058] To accelerate the model, multiple segments to be predicted are concatenated after the segmented text sentence. The segments to be predicted share positional information with the segments in the text sentence. Since the preprocessing language model can only process sentences of finite length at a time, only m segments with similar positions are selected as segments to be predicted and concatenated at the end of the sentence to form a new sample.

[0059] Specifically, it includes:

[0060] Step 11: First, segment the text statement into words. Then, add the [CLS] symbol to the segmented text statement to obtain the sequence {[CLS], T1, T2, T3, T...}. i ..., T n}; where T i This is represented as a token obtained after word segmentation of a text statement.

[0061] Step 12: Combine the m segments to be predicted from the end of the text statement, represented as {[CLS]; T1, T2, T3, ..., T...} n ;S1, S1,...,S1;S1,S2,S3,...,S m}, {S1, S1, ..., S1} represents the first position information of the added segment to be predicted, {S1, S2, S3, ..., S1} represents the first position information of the added segment to be predicted. m} represents the tail position information of the segment to be predicted, until all segment position information S1 to S2 is traversed. n The position information of the added segment to be predicted shares the position information with the corresponding word unit in the text, resulting in a total of Z spliced ​​segments to be predicted;

[0062] The formula for calculating Z is:

[0063]

[0064] Where L represents the length of the segment to be predicted, and N represents the number of words in the segmented text. The length L of the segment to be predicted is less than the length N of the text. The sample sentences input into BERT are constructed as follows: Figure 2 As shown in the figure, the maximum length of the segment to be predicted is set to 4. The maximum length can be customized according to the specific application scenario, which means that the maximum length of an entity segment does not exceed 4. The maximum length can be adjusted according to the actual situation, and adjacent segments are combined.

[0065] Step 13: Construct entity labels and relationship labels. Entity labels consist of entity boundary (beginning and end) information and entity type label information. Relationship labels consist of the boundary (beginning and end) information of the subject-object entity pair and relationship type label.

[0066] Step 2: Input the sample sentences into the pre-trained BERT model to obtain the word vectors and [CLS] sentence vectors generated by the BERT model after encoding;

[0067] Step 3: Construct task feature vectors related to the Entity Recognition (NER) task. Where i and j represent the first and last labels of the predicted entity segment;

[0068] Step 31: Perform max pooling on the word vector feature information to obtain h. m The calculation formula is:

[0069] h m = maxpooling(h1,h2,...,h n );

[0070] Step 32: Concatenate partial feature vectors to obtain task features. The calculation formula is:

[0071]

[0072] Where cat represents the concatenate operation, h i h represents the first position information feature of the predicted entity segment. j This represents the tail position information feature of the predicted entity segment. This represents the first position information feature of the segment to be predicted added at the end of the sample. This represents the tail position information features of the segment to be predicted added to the end of the sample. By combining these features, m candidate segments can be identified at once, such as... Figure 3 As shown;

[0073] Step 33, The data is fed into the NER classifier to obtain the prediction result for entity type k. The formula is:

[0074]

[0075] Among them, W e b e Let represent the trainable parameters of the task model for entity extraction and relation extraction, where k represents the type of entity, and ε represents the set of entity types. Then add the result to set E;

[0076] Step 34: Calculate the cross-entropy loss of the NER part. The formula is:

[0077]

[0078] Where N represents the number of samples, y ij Indicates whether it is the current category;

[0079] Step 4: Construct task feature vectors related to the relation extraction (RE) task. Where a and b represent the subject fragment a to be predicted and the object fragment b to be predicted, fragments a and b come from set E, and samples containing only fragment b are selected from the fragments to be predicted that are spliced ​​at the end of the text statement.

[0080] Step 41: Concatenate partial feature vectors to obtain task features. The calculation formula is

[0081]

[0082] Where cat represents the concatenate operation, and h0 represents the [CLS] sentence feature vector output by the pre-trained language model; This is represented by the first position feature of the main segment a. This is represented by the tail position feature of the main segment a. This is represented by the first position feature of the candidate object segment. Represented as the tail position feature of the candidate object fragment, such as Figure 4 As shown;

[0083] Step S42, will The data is fed into the RE classifier, and the predicted relationship type between subject fragment a and object fragment b is 'l'. The formula is:

[0084]

[0085] Among them, W r b r Let represent the trainable parameters of the model, and l represent the type of relation. Represented as a set of relation types;

[0086] Step 43: Calculate the cross-entropy loss of the RE component. The formula is:

[0087]

[0088] Where N represents the number of samples, y ab Indicates whether it is the current category;

[0089] Step 5: Calculate the loss obtained in step 34. The loss obtained in step 43 The total loss of the task models for entity recognition and relation extraction is obtained by summing the coefficients. The formula is:

[0090]

[0091] Where α and β are dynamic weights. As the training process progresses, the weight α gradually decreases and the weight β gradually increases. The model is trained using the final loss, and the Adam optimizer is selected to optimize the model parameters.

[0092] Step 6: Save the model that performs best on the validation set and test it on the samples to be predicted to obtain the final test results. Apply a threshold filter to the results obtained in step 33; a result greater than 0.5 indicates that the predicted entity fragment belongs to a certain entity type. Apply a threshold filter to the results obtained in step 42; a result greater than 0.5 indicates that predicted fragments a and b belong to a certain relation label type. Combine the prediction results to obtain the final triplet.<s,r,o> .

[0093] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A method for entity recognition and relation extraction that integrates lexical boundaries and semantic information, characterized in that, Includes the following steps: Step 1: Construct sample inputs and labels for the preprocessed language model; Step 2: Input the samples into the pre-trained BERT model, and the BERT model outputs the feature vector of the last layer; Step 3: Construct the task feature vector for entity recognition. The feature vector is obtained by concatenating the features after max pooling of the sentence, the predicted segment boundary word features, and the boundary features concatenated at the end of the sample. The data is fed into a NER classifier to obtain the classification result and the loss is calculated. ; Step 3 in detail include: Step 31: Perform max pooling on the word vector feature information to obtain... The calculation formula is: ； Step 32: Concatenate the partial feature vectors to obtain the task feature vector. The calculation formula is: ， Here, cat represents the concatenate operation. This represents the first position information feature of the predicted entity segment. This represents the tail position information feature of the predicted entity segment. This represents the first position information feature of the segment to be predicted added at the end of the sample. The tail position information features of the segment to be predicted are added to the end of the sample, and m candidate segments are identified by combining them at once; Step 33, The data is fed into a NER classifier, and the entity type is obtained as follows: Prediction results The formula is: ， in, These represent the trainable parameters of the task models for entity extraction and relation extraction. Represented as the type of the entity, Represented as a collection of entity types; Step 34: Calculate the cross-entropy loss of the NER part. The formula is: ， in, Indicates whether it is the current category; Step 4: Construct task feature vectors related to the relation extraction task. The result is obtained by concatenating the sentence vector, the boundary features of the subject segment to be predicted, and the boundary features of the object segment to be predicted. This result is then fed into the RE classifier to obtain the classification result and calculate the loss. ; Step 5: Determine the losses With loss The total loss is obtained by adding the coefficients together. ; Step four specifically includes: Step 41: Concatenate some feature vectors to obtain the task feature vector. The formula is: ]， Here, `cat` represents the `concatenate` operation. This is represented as the [CLS] sentence feature vector output by the pre-trained language model; This is represented by the first position feature of the main segment a. This is represented by the tail position feature of the main segment a. This is represented by the first position feature of the candidate object segment. This is represented as the tail position feature of the candidate object segment; Step S42, will The data is fed into the RE classifier, and the relationship type between subject fragment a and object fragment b is obtained as follows: Prediction results The formula is: ， in, These are the parameters that the model can train. Represented as the type of relation. Represented as a set of relation types; Step 43: Calculate the cross-entropy loss of the RE component. The formula is: ， in, This indicates that the text statement after word segmentation contains [a certain number of words]. n Each word element, Indicates whether it is the current category; Step 6: Combine the results of entity recognition and relation extraction to obtain triples.

2. The entity recognition and relation extraction method integrating lexical boundaries and semantic information according to claim 1, characterized in that, Step one specifically includes: Step 11: Segment the text statement into words, and add the [CLS] symbol to the segmented text statement to obtain the sequence {[CLS], T1, T2, T3, T...} i , ..., T n }; where T i This is represented as a token obtained after word segmentation of a text statement. Step 12: Combine the m segments to be predicted from the end of the text statement, represented as {[CLS]; T1, T2, T3, ..., T...} n ;S1, S1,...,S1;S1,S2,S3,...,S m }; where {S1, S1, ..., S1} represents the first position information of the added segment to be predicted, {S1, S2, S3, ..., S1} represents the first position information of the added segment to be predicted. m } represents the tail position information of the segment to be predicted, until all segment position information S1~S2 is traversed. n The position information of the added segment to be predicted shares the position information with the corresponding word unit in the text, resulting in a total of Z spliced ​​segments to be predicted; Step 13: Construct entity labels and relationship labels. Entity labels consist of entity boundary information and entity type label information. Relationship labels consist of the boundary information of the subject-object entity pair and the relationship type label.

3. The entity recognition and relation extraction method integrating lexical boundaries and semantic information according to claim 2, characterized in that, The formula for calculating the Z-segment segment to be predicted is: ； Where L represents the length of the segment to be predicted.

4. The entity recognition and relation extraction method integrating lexical boundaries and semantic information according to claim 1, characterized in that, In step two, the feature vectors include word vectors and sentence vectors.

5. The entity recognition and relation extraction method integrating lexical boundaries and semantic information according to claim 1, characterized in that, Total loss in step five The formula is: ， Where α and β are dynamic weights.

Images