A method and device for training a named entity and relationship joint recognition model

Abstract

The application relates to a training method and device of a named entity and relationship joint recognition model, and belongs to the technical field of natural language processing; the method solves the problem that in the prior art, named entity and relationship recognition needs to be performed in two independent tasks, namely, entity recognition and classification, and the time consumption is long, and the resource use efficiency is reduced; the model training method comprises the following steps: obtaining a named entity and relationship joint annotation data set D ALL , adding task descriptions related to named entity recognition and relationship recognition to original text data in the data set D ALL , constructing target outputs according to the task descriptions, further constructing a training sample set, training data in the training sample set, and obtaining a named entity and relationship joint recognition model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to a named entity and relation joint recognition model training method and computer equipment. BACKGROUND

[0002] In recent years, with the rapid development of the Internet, we have ushered in the era of information explosion, and every day there are countless news, entertainment, advertising and other information being generated and rapidly spreading. In order to better manage and extract useful information from the vast amount of information, information extraction technology based on natural language processing has emerged. Named entity recognition (NER) and relation extraction (RE) are two crucial steps in the information extraction process. Named entity recognition is to identify entities with specific meanings in the text, mainly including names, place names, organization names, and proper nouns. Relation extraction is to extract triples (subject, object, relation) from a text, also known as triple extraction. Named entity recognition and relation extraction play a crucial role in text search, knowledge graph construction, and machine intelligence question answering.

[0003] Currently, the named entity recognition methods mainly include sequence labeling and pointer network. The sequence labeling method refers to predicting the label of the input text word by word, and the label contains whether the word belongs to an entity and which type of entity it belongs to. The classic sequence labeling methods are BIO, BMES, and BIOSE. The mainstream named entity recognition model based on sequence labeling is BERT+CRF (Conditional Random Field). CRF supports conditional constraints when decoding, which ensures the completeness and logicality of the recognized entities to some extent. However, since it is decoded word by word, it cannot be parallel computed, so the decoding speed is slow. The pointer network predicts the head pointer and tail pointer of all possible entities in the input text in all category dimensions, and uses these position pointers to find the corresponding entities. The pointer network can be parallel computed when decoding, so the prediction speed is faster than sequence labeling. However, neither sequence labeling nor pointer network is ideal in identifying long entities and nested entities.

[0004] Currently, the relation extraction models are mainly divided into pipeline models (Pipline) and joint models (Joint). The pipeline model first extracts the subject and object of the input text, and then determines the relationship. The joint model simultaneously extracts the subject and object and classifies the relationship. The pipeline model generally consists of two models, and the entire recognition process is performed in steps, so the model is flexible. However, the errors in entity extraction will affect the performance of relationship extraction. The joint model usually extracts relationships through shared parameters or constructing specific decoding methods. The model structure is more complex, but the effect is better.

[0005] From the perspective of technical implementation, the current named entity recognition and relationship extraction model needs to rely on a complex decoding process to obtain the final result, and such decoding process often cannot utilize the GPU of the computer for operation, and is time-consuming. Moreover, the entity is identified first and then classified, and the two independent tasks are divided to reduce the resource utilization efficiency, but there is no complete method to simultaneously realize the two tasks of named entity recognition and relationship extraction. SUMMARY

[0006] In view of the above analysis, the present application aims to provide a named entity and relationship joint recognition model training method and device; solve the problem of named entity and relationship recognition in the prior art, which needs to identify the entity first and then classify, and divides into two independent tasks for recognition, which is time-consuming and reduces the resource utilization efficiency.

[0007] The purpose of the present application is mainly realized by the following technical solutions:

[0008] On the one hand, the present application provides a named entity and relationship joint recognition model training method, comprising the following steps:

[0009] Obtain a named entity and relationship joint annotation data set D ALL , wherein each piece of data in the data set D ALL includes at least one independent entity and one relationship;

[0010] Add a task description related to named entity recognition and relationship recognition to the original text data in the data set D ALL , and construct a target output according to the task description;

[0011] Based on the original text in the data set D ALL , the corresponding task description and the target output, a training sample set is constructed, and the data in the training sample set is used for training, and the named entity and relationship joint recognition model is obtained through loss function iteration update.

[0012] Further, the named entity and relationship joint annotation data set D ALL is obtained, comprising:

[0013] Respectively obtain a named entity recognition data set and a relationship extraction data set;

[0014] Supplement the relationship annotation to the original text in the named entity recognition data set to obtain a named entity recognition data set D NER+RE supplemented with relationship annotation; the data set D NER+RE includes relationship-annotated data and non-relationship-annotated data;

[0015] Supplement the original text in the relation extraction dataset with named entity annotation; obtain the relation extraction dataset D supplemented with named entity annotation RE+NER The dataset D RE+NER includes independent entity annotated data and non-independent entity annotated data

[0016] The non-relation annotated data in the dataset D NER+RE and the non-independent entity annotated data in the dataset D RE+NER are randomly merged to obtain the dataset D 3 .

[0017] The relation annotated data in the dataset D NER+RE , the independent entity annotated data in the dataset D RE+NER and the dataset D 3 are merged to obtain the dataset D ALL .

[0018] Further, the named entity recognition dataset and the relation extraction dataset are trained respectively to obtain a named entity recognition model and a relation extraction model

[0019] All data in the relation extraction dataset are predicted by using the named entity recognition model to obtain the named entity annotation of the relation extraction dataset; obtain the relation extraction dataset D supplemented with named entity annotation RE+NER .

[0020] All data in the named entity recognition dataset are predicted by using the relation extraction model to obtain the named entity recognition dataset D supplemented with relation annotation NER+RE .

[0021] Further, the task description corresponds to a task including: entity relation simultaneous recognition, named entity recognition and relation recognition

[0022] The task description of the entity relation simultaneous recognition is to simultaneously recognize all entities, entity categories and all relations in the original text

[0023] The task description of the named entity recognition is to recognize all entities and entity categories in the original text

[0024] The task description of the relation recognition is to recognize all relation triplets in the original text; the relation triplet includes the relations existing in the original text and the subject and object corresponding to each relation

[0025] Further, for the task of entity relation simultaneous recognition, the target output is constructed by the following way

[0026] Entity conversion: traverse all entity categories of each original text, get the entities in each entity category, combine each entity in the form of "entity + is + entity category" to form a sentence, and concatenate each entity sentence with ",";

[0027] Relationship conversion: traverse all relationship labels of each original text, get the relationship in each relationship label and the subject and object corresponding to each relationship, combine each relationship in the form of "subject + is + relationship + is + object" to form a sentence, and concatenate each relationship sentence with ",".

[0028] The converted entity sentence and relationship sentence corresponding to each original text are concatenated with "," to construct the target output corresponding to each original text.

[0029] Further, it also includes auxiliary tasks: entity extraction, entity classification, relationship subject and object extraction, and relationship classification, wherein:

[0030] The task description of entity extraction is to extract all entities in the original text.

[0031] The task description of entity classification is to identify the entity category corresponding to the entity in the original text.

[0032] The task description of relationship subject and object extraction is to identify the subject and object of the relationship in the original text.

[0033] The task description of relationship classification is to identify the relationship between the subject and the object in the original text.

[0034] Further, the data in the data set D ALL is divided into 7 parts according to the preset proportion, and the corresponding target output is constructed according to the seven task descriptions. After the construction is completed, the data is merged and shuffled to construct the training sample set.

[0035] Further, the named entity and relationship joint recognition model is obtained by training based on a T5 pre-training model, including an encoder and a decoder.

[0036] According to the original text and the task description, the target output is predicted by calculating through a self-attention mechanism and a feedforward network, and the converged model is obtained through loss iteration, which is the named entity and relationship joint recognition model.

[0037] Further, the loss iteration is performed by the following formula:

[0038]

[0039] wherein, is the predicted probability of time step t, is the true probability of time step t.

[0040] The total loss Loss is obtained by adding the losses of all time steps T :

[0041]

[0042] Wherein, T is all time steps.

[0043] In another aspect, a computer device is also provided, comprising at least one processor, and at least one memory connected with the processor in communication;

[0044] The memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the training method of the named entity and relationship joint recognition model.

[0045] The beneficial effects of the technical solution are:

[0046] 1. The application regards the named entity recognition and relationship extraction as a text generation task, constructs training samples according to task description and corresponding target output, and trains to obtain a named entity and relationship joint recognition model. Compared with other models, the recognition model can recognize named entities and relationships simultaneously, has the advantages of simple decoding process and fast prediction speed.

[0047] 2. The application constructs seven different training data to train the model simultaneously, including three main tasks and four auxiliary tasks. The tasks are distinguished by task description. In actual application, seven different tasks can be completed by changing the task description, and the application has great practical value.

[0048] Other features and advantages of the application will be set forth in the following description of the application, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the application. The objects and other advantages of the application can be achieved and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0049] The accompanying drawings are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the principles of the application. In the drawings:

[0050] Figure 1 The named entity and relationship joint recognition model training method flowchart of the embodiments of the application.

[0051] Figure 2 The model training method flowchart of the embodiments of the application.

[0052] Figure 3A cross-supplement data set flow schematic diagram of an embodiment of the present application;

[0053] Figure 4 A final training data construction flow schematic diagram of an embodiment of the present application;

[0054] Figure 5 A T5 model overall structure schematic diagram of an embodiment of the present application; DETAILED DESCRIPTION

[0055] The preferred embodiments of the present application will be described in detail below with reference to the drawings, wherein the drawings constitute a part of this application and are used to explain the principles of the embodiments of the present application, but are not used to limit the scope of the present application.

[0056] The present application discloses a training method of a named entity and relationship joint recognition model, which takes the named entity recognition and relationship extraction two tasks as a text generation task, converts the labels of the named entity and relationship in the original text into a format of word description as the target output text of the model, so that the model can recognize all entities and relationships in the original text at one time. In order to improve the accuracy of model recognition, four auxiliary tasks are added for data enhancement training, and descriptions of task requirements are added in the training data to distinguish each task, and finally a unified model which can complete the named entity and relationship extraction tasks at the same time is obtained.

[0057] The training method of the named entity and relationship joint recognition model in the embodiment, as shown in Figure 1 , comprises the following steps:

[0058] Step S1, obtaining a data set D of named entity and relationship joint annotation ALL , wherein each piece of data in the data set D ALL includes at least one independent entity and one relationship;

[0059] Specifically, a named entity recognition data set and a relationship extraction data set are obtained respectively; the named entity recognition data set and the relationship extraction data set are used for training respectively to obtain a named entity recognition model and a relationship extraction model;

[0060] The named entity recognition model is used to predict all data in the relationship extraction data set to obtain the named entity annotation of the relationship extraction data set; a relationship extraction data set D RE+NER with supplemented named entity annotation is obtained; the data set D RE+NER includes data with independent entity annotation and data without independent entity annotation;

[0061] The relationship extraction model is used to predict all data in the named entity recognition data set to obtain a named entity recognition data set D NER+REThe data set D NER+RE includes data with relationship annotation and data without relationship annotation;

[0062] The data set D NER+RE includes data without relationship annotation and the data set D RE+NER includes data without independent entity annotation. 3 ;

[0063] The data set D NER+RE includes data with relationship annotation, the data set D RE+NER includes data with independent entity annotation, and the data set D 3 . ALL .

[0064] Preferably, the named entity recognition data set can be selected from a People's Daily data set and a cluener data set, and the People's Daily data set and the cluener data set already contain named entity annotation; the relationship extraction data set can be selected from a lic2021 data set, and the lic2021 data set already contains relationship triple annotation. The named entity recognition data set and the relationship extraction data set are consistent in language, and the contents of the data sets are as similar as possible in the field to ensure stable prediction effect of the model.

[0065] First, the data in the data set is denoised: the special symbols and redundant spaces in the above data set that affect the model training are deleted, all full-width characters are converted to half-width characters, all traditional Chinese characters are converted to simplified Chinese characters, and all uppercase English letters are converted to lowercase. The denoised named entity data set is denoted as D NER , and the denoised relationship extraction data set is denoted as D RE .

[0066] Since the final model training needs to use two data sets at the same time, and the two data sets are similar in the field, there may be some unannotated relationships similar to the relationship extraction data set in the named entity recognition data set, and for the same reason, there may be unannotated entities similar to the named entity recognition data set in the relationship extraction data set, that is, simply merging the data sets will cause data leakage, so the annotations of the two data sets need to be cross-supplemented.

[0067] Cross-supplementing the data sets needs to supplement the annotation of the relationship in the named entity recognition data set D NER and supplement the annotation of the named entity in the relationship extraction data set D RE , so that the two data sets have annotations of both named entities and relationships, and prepare for the subsequent model training.

[0068] The above steps of cross-supplementing the data sets Figure 2As shown, it includes:

[0069] Choose a named entity recognition model and a relation extraction model respectively; the choice of model is optional and not strictly required. This example uses the BERT+CRF model for named entity recognition and the BERT+CNN model for relation extraction. Specifically, the selected model must support the language of the dataset.

[0070] A named entity recognition model is trained using a named entity recognition dataset. The trained model is denoted as M. NER Train a relation extraction model using the relation extraction dataset; the trained model is denoted as M. RE ;

[0071] Using the trained named entity recognition model M NER Predicting relation extraction dataset D RE From all the data, obtain the named entity annotation L for the relation extraction dataset. NER ; mark named entities as L NER Add to the original relation extraction dataset D RE In this process, we obtained a supplementary dataset D for relation extraction based on named entity annotations. RE+NER ;

[0072] Using the trained relation extraction model M RE Predicting Named Entity Recognition Dataset D NER From all the data, obtain the relation annotation L of the named entity recognition dataset. RE Label the relationship L RE Added to the original named entity recognition dataset D NER In this process, a named entity recognition dataset D with supplementary relation annotations was obtained. NER+RE ;

[0073] D RE+NER The data with and without independent entity labels were separated to obtain the dataset with independent entity labels. and datasets without independent entity annotations D NER+RE The data with and without relational labels were separated to obtain the dataset with relational labels. And unrelated labeled datasets Horizontal Merging and We obtain a dataset D that includes both independent entities and relation annotations. 3 ;unified D 3 Annotation format; merge D 3 The merged dataset is obtained, denoted as D.ALL .

[0074] It should be noted that the D RE+NER Each piece of data in the theory should be increased in named entity annotation, because the original data set is annotated with triplets such as (subject, object, relationship), and the subject and object are entities that can be identified by M NER The difference lies in whether there is an independent entity in the original text. Independent entity refers to an entity that cannot form a relationship with other entities in the text. In order to make the named entity and relationship joint recognition model obtained after training can identify entities with and without relationships, the training data needs to meet the condition that each piece of data contains at least one relationship and at least one independent entity.

[0075] For example, "2009" in the sentence in Table 1 is an independent entity, so the data containing independent entities such as the sentence in Table 1 are classified into the D .

[0076] Table 1 Relationship extraction data set with independent entity annotation examples

[0077]

[0078]

[0079] The rest of the data without independent entity annotation refers to the identified entities containing only the subject and object in the original relationship, such as the sentence in Table 2. Such sentences are classified into the D

[0080] Table 2 Relationship extraction data set without independent entity annotation examples

[0081]

[0082] Similarly, a part of the D NER+RE The sentence in Table 3 has both entity and relationship annotations, and such sentences are classified into the D .

[0083] Table 3 Examples of adding relationship annotations to entity identification data sets

[0084]

[0085] And the other part of the data has not been added relationship annotation due to the absence of relationship, for example, the sentence in Table 4 has no relationship description except "reporter" is a professional entity, such sentences are classified into the D

[0086] Table 4 Entity identification data set without relationship annotation examples

[0087]

[0088] By analyzing the dataset and The data is merged to obtain training data that meets the training conditions and contains at least one relation and at least one independent entity. and The datasets have been filtered and found to meet the criteria, so they have been merged. and D 3 The dataset D that meets the conditions is obtained. ALL .

[0089] For example, the sentences in Table 2 belong to the dataset. The data in Table 4 contains sentences from a dataset without independent entity annotations. For data without relational annotations, these two types of sentences are horizontally merged, i.e., sentence-to-sentence merging, entity-to-entity merging, and relation-to-relation merging. Taking the sentences in Tables 2 and 4 as examples, the annotation after merging is shown in Table 5. The merged result is the independent entity "Reporter".

[0090] Table 5 shows the horizontal merging of data with no independent entity annotations and data with no relationship annotations.

[0091]

[0092] Since named entity recognition and relation extraction tasks rarely rely on the coherence between sentences, direct concatenation in this way will not affect the training effect. and The number of sentences in the dataset is usually inconsistent, so multiple sentences can be concatenated, based on the same principle as described above. The concatenated dataset D 3 Each data point contains at least one relation and at least one independent entity, satisfying the training data requirements.

[0093] For dataset D 3 Before merging, the data in the dataset needs to be labeled in a unified format. This example uses JSON format, where each text and its corresponding tag constitute a JSON dictionary. The internal structure of the JSON dictionary is shown in Table 6.

[0094] Table 6 shows the JSON format structure of the data annotations.

[0095]

[0096] Wherein, the English keys "text", "entity_label", "relation_label", "head", "end", "relation" in the dictionary are fixed and unchanged, and the rest of the Chinese keys and values need to be modified and added or reduced according to the specific content. For example, the json format of the sentence in Table 5 is shown in Table 7:

[0097] Table 7 Data annotation example

[0098]

[0099] Preferably, the data set D 3 is merged by longitudinal merging, that is, the data sets are spliced vertically to obtain the spliced data set D ALL .

[0100] Step S2: adding task description related to named entity recognition and relation recognition to the original text data in the data set D ALL , and constructing target output according to the task description;

[0101] Specifically, the model training includes 3 main tasks and 4 auxiliary tasks, so 7 kinds of data annotation are needed. The data in D ALL is divided into seven parts in the ratio of 2:2:2:1:1:1:1, which are used to construct main task 1-3 and auxiliary task 1-4, respectively. Each main task data accounts for 20% of the total data, and auxiliary task data accounts for 10% of the total data. After construction, the data is merged and shuffled to divide into training set, validation set and test set in the ratio of 8:1:1. The overall process of constructing the final training data is shown in Figure 3 . Each piece of data includes three parts: original text, task description, and target output.

[0102] Among them, the main tasks include: entity relation simultaneous recognition, named entity recognition and relation recognition; the auxiliary tasks include: entity extraction, entity classification, relation subject and object extraction and relation classification. Specifically:

[0103] 1) Main task 1 - entity relation simultaneous recognition

[0104] The goal of main task 1 is to let the model recognize all entities, corresponding entity categories and all relations in the original text at the same time, and the target output is the text representation of all entity, entity category and relation triplets in the text. The process of converting the structure shown in Table 6 into the target output of the main task is as follows:

[0105] For the conversion of entities, first traverse all entity categories in each original text, and then traverse the entities in each entity category. For each entity, combine it into a sentence in the form of "entity + is + entity category", and concatenate all entity sentences with ",".

[0106] For the conversion of relations, traverse all "relation_label" values in each original text that represent relations, and for each dictionary, combine it into a sentence in the form of "subject + of + relation + is + object", and concatenate all relation sentences with ",".

[0107] Finally, concatenate all entity sentences and relation sentences in each original text with "," as the target output.

[0108] For example, the training data of main task 1 corresponding to the data in Table 7 is shown in Table 8:

[0109] Table 8 Training data example of main task 1

[0110]

[0111] 2) Main task 2 - named entity recognition

[0112] The goal of main task 2 is to let the model perform named entity recognition, that is, to identify all entities and entity categories in the original text. The target output is the text representation of all entities and entity categories in the text, and the conversion method is the same as that of main task 1 for entity conversion.

[0113] For example, the training data of main task 2 corresponding to the data in Table 7 is shown in Table 9:

[0114] Table 9 Training data example of main task 2

[0115]

[0116] 3) Main task 3 - relation triple recognition

[0117] The goal of main task 3 is to let the model perform relation triple recognition, that is, to identify all existing relations and their subjects and objects in the original text. The target output is the text representation of all relation triples in the text, and the conversion method is the same as that of main task 1 for relation triple conversion.

[0118] For example, the training data of main task 3 corresponding to the data in Table 7 is shown in Table 10:

[0119] Table 10 Training data example of main task 3

[0120]

[0121] 4) Auxiliary task 1 - entity extraction

[0122] The goal of auxiliary task 1 is to enable the model to extract all entities from the original text, excluding entity categories. This trains the model's ability to recognize entities, and the target output is all entities in the original text. The process of transforming the structure shown in Table 6 into the target output is as follows:

[0123] First, iterate through all entity categories in each original text. Then, iterate through the entities in each entity category and concatenate all entities using commas.

[0124] For example, the training data for auxiliary task 1 corresponding to the data in Table 7 is shown in Table 11:

[0125] Table 11 Example of training data for auxiliary task 1

[0126]

[0127] 5) Auxiliary Task 2 – Entity Classification

[0128] The goal of auxiliary task 2 is to enable the model to classify entities in the original text. The task description is to identify the entity category corresponding to the entity in the original text, which trains the model's entity classification ability. Auxiliary task 2 requires the model to provide all entities in the original text in the task description and the category corresponding to these entities in the target output.

[0129] Taking the structure shown in Table 6 as an example, first, all entity categories are traversed; then, the entities within each entity category are traversed, and all entities are concatenated together with commas to form the content of the task description; finally, all entity categories are concatenated together with commas to form the content of the target output. The order of the task description and the target output should correspond consistently.

[0130] For example, the training data for auxiliary task 2 corresponding to the data in Table 7 is shown in Table 12:

[0131] Table 12 Examples of training data for auxiliary task 2

[0132]

[0133]

[0134] 6) Auxiliary Task 3 – Relationship Subject-Object Extraction

[0135] The goal of Auxiliary Task 3 is to enable the model to identify the subject and object of the relationships in the original text. The task description is to identify the subject and object of the relationships in the original text, which trains the model's ability to identify subjects and objects. When constructing the task description, Auxiliary Task 3 needs to provide all the relationships in the original text in the task description, and provide the subjects and objects corresponding to these relationships in the target output.

[0136] Taking the structure shown in Table 6 as an example, the dictionary representing the relationship in all values of "relation_label" is traversed, and the "relationship" in all dictionaries is spliced together with "," as the content in the task description; for each dictionary, it is combined into a sentence in the form of "subject + and + object", and all combined sentences are spliced together with "," as the content of the target output. The order of the task description and the target output content should correspond.

[0137] For example, the training data of the auxiliary task 3 corresponding to the data in Table 7 is shown in Table 13:

[0138] Table 13 Example of training data of auxiliary task 3

[0139]

[0140] 7) Auxiliary task 4 - relationship classification

[0141] The goal of auxiliary task 4 is to let the model classify the relationship between the subject and the object in the original text, and the task description is to identify the relationship between the subject and the object in the original text. The ability of the model to classify relationships is trained, and auxiliary task 4 needs to give all the subjects and objects in the original text in the task description when constructing the task description. The target output is all the relationships existing in the text. The task description and the target output content of auxiliary task 4 are just the opposite of auxiliary task 3, and the specific conversion method can refer to auxiliary task 3.

[0142] For example, the training data of the auxiliary task 4 corresponding to the data in Table 7 is shown in Table 14:

[0143] Table 14 Example of training data of auxiliary task 4

[0144]

[0145] Step S3: Based on the original text and the corresponding task description and target output in the data set D ALL , a training sample set is constructed, and the data in the training sample set is used for training, and the loss function is iteratively updated to obtain a named entity and relationship joint recognition model;

[0146] The named entity and relationship joint recognition model is used to predict the output text corresponding to the task description of the to-be-recognized text according to the input to-be-recognized text and the task description.

[0147] Specifically, the seven methods described above are used to process the data in the data set D ALL , and the task description is directly connected to the original text as the original text, and a new data set D ALL_ is obtained after processing. ALL_The data in the middle is disordered, and is divided into a training set, a validation set and a test set in a ratio of 8:1:1, denoted as D train , D vaild , D test The final training data is constructed.

[0148] The recognition model of the embodiment selects a T5 pre-training model, and is trained in a pre-training + fine-tuning manner. The input of the model is: original text + task description, and the target output is D train All data are used to train the model, and the specific process is as follows:

[0149] First, the T5 pre-training weight is loaded, and the pre-training weight can be updated in the training stage. The T5 is composed of 12 same encoders and 12 same decoders, and the structure is as shown in Figure 4 Each encoder and decoder is calculated by a self-attention mechanism and a feedforward network inside, and the output of the encoder is needed when the decoder is calculated.

[0150] The first encoder of the T5 receives the original text + task description as the input X input , and obtains a word vector set with a length of n and a dimension of 768 through word embedding and position encoding n is the token number after segmentation.

[0151] H 1 The output hidden vector of the first encoder is obtained through the multi-head self-attention and the feedforward network layer as the input of the second encoder. Among them, the number of heads of the multi-head self-attention is 12, and the calculation formula of each head is:

[0152] Q=H i ·W q , K=H i ·W k , V=H i ·W v (i∈[1,12])

[0153]

[0154] H i is the input hidden vector of the i-th layer, W q , W k , W v are three trainable matrices, which are randomly initialized during training, and have a shape of 768x n, d k =64.

[0155] The output of the 12th encoder is obtained by repeating the calculation for 12 times When decoding, the self-attention mechanism of each decoder is calculated as follows:

[0156] Q = H e · W q , K = H e · W k , (i e [1, 12])

[0157]

[0158] where Q, K are calculated using the encoder output H e , V is calculated using the decoder hidden vector of the i-th layer W q , W k , W v are three trainable matrices, randomly initialized during training, with shape 768 x n, d k = 64.

[0159] At each decoding time step t, the loss Loss between the predicted probability and the true probability t is calculated using cross-entropy, and the loss of all time steps is added to obtain the total loss Loss T :

[0160]

[0161] The loss is backpropagated, and the model parameters are updated by minimizing Loss T The model is trained for 50 rounds using all data, and after each round, the validation set D vaild is used for validation, and the model with the best performance is saved as the final named entity and relation joint recognition model.

[0162] Another embodiment of the application provides a computer device, comprising at least one processor and at least one memory communicatively connected to the processor.

[0163] The memory stores instructions executable by the processor, and the instructions are used to implement the model training method of the foregoing embodiments when executed by the processor.

[0164] In summary, the training method of the named entity and relation joint recognition model provided by the embodiments of the application can simultaneously complete the named entity recognition and relation extraction tasks, and can selectively complete one of the tasks, and has strong practicability and flexibility. Figure 5As shown, the method of the present application can be divided into two parts: constructing training data and training model. Among them, the construction of training data required is the focus of the implementation of the method, including three steps of obtaining and preprocessing data set, cross supplementing data set, constructing final training data, and the model obtained by training model is the final named entity and relationship joint recognition model of the method. The present application takes the named entity and relationship extraction task as a text generation task, trains the text generation model by constructing the training data in the form of text, and uses three main tasks and four auxiliary tasks when training the model. By giving different task descriptions, the data and tasks are distinguished, the performance and prediction accuracy of the model are improved, and finally the joint recognition model which can complete the named entity and relationship extraction task at the same time is obtained.

[0165] The training method of the named entity and relationship joint recognition model of the present application breaks the traditional training method, takes the named entity recognition and relationship extraction two tasks as a text generation task, and uses the default decoding process of T5 model to decode the final result. Compared with the traditional sequence labeling and pointer network method, the time of custom decoding is greatly saved. In addition, the model of the present application also supports changing the output result through task description to meet the needs of various subtasks of named entity and relationship, and has strong practicability and flexibility.

[0166] Those skilled in the art can understand that all or part of the processes of the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the program can be stored in a computer readable storage medium. Among them, the computer readable storage medium is a disk, an optical disk, a read-only memory or a random access memory, etc.

[0167] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A method for training a named entity and relation joint recognition model, characterized in that, The method comprises the following steps: Obtain a dataset D of joint annotation of named entity and relation ALL , comprising: respectively obtaining a named entity recognition dataset and a relation extraction dataset; supplementing the original text in the named entity recognition dataset with relation annotation to obtain a named entity recognition dataset D supplemented with relation annotation NER+RE ; the dataset D NER+RE includes data with relation annotation and data without relation annotation; supplementing the original text in the relation extraction dataset with named entity annotation; obtaining a relation extraction dataset D supplemented with named entity annotation RE+NER , the dataset D RE+NER includes data with independent entity annotation and data without independent entity annotation; randomly merging the data without relation annotation in the dataset D NER+RE and the data without independent entity annotation in the dataset D RE+NER to obtain a dataset D 3 ; merging the data with relation annotation in the dataset D NER+RE , the data with independent entity annotation in the dataset D RE+NER and the dataset D 3 to obtain the dataset D ALL ; wherein each piece of data in the dataset D ALL includes at least one independent entity and one relation; adding task descriptions related to named entity recognition and relationship recognition to original text data in the data set D ALL , and constructing target outputs according to the task descriptions; Based on the data set D ALL The original text in the data set D, the corresponding task description and the target output are used to construct a training sample set. The data in the training sample set is used for training. Through loss function iterative updating, a named entity and relationship joint recognition model is obtained. The named entity and relationship joint recognition model is obtained by training a pre-training model based on T5 and includes an encoder and a decoder. According to the original text and the task description, the target output is predicted through self-attention mechanism and feedforward network calculation. Through loss iteration, a converged model, i.e. the named entity and relationship joint recognition model, is obtained. 2.The method of claim 1, wherein, The named entity recognition dataset and the relation extraction dataset are used for training respectively to obtain a named entity recognition model and a relation extraction model; The named entity recognition model is used to predict all data in the relation extraction data set, to obtain named entity annotation of the relation extraction data set; and a relation extraction data set D with supplementary named entity annotation is obtained RE+NER ; The relationship extraction model is trained by using the annotated named entity recognition data set D and the annotated relationship data set D. NER+RE . 3.The method of claim 1, wherein, The task description corresponds to a task comprising entity relation simultaneous recognition, named entity recognition, and relation recognition; The task description of the entity relation simultaneous recognition is to simultaneously recognize all entities, entity categories, and all relations in the original text; The task description of the named entity recognition is to recognize all entities and entity categories in the original text; The task description of the relation recognition is to recognize all relation triples in the original text; the relation triple comprises a relation existing in the original text and a subject and an object corresponding to each relation. 4.The method of claim 1, wherein, For the task of entity relation simultaneous recognition, the target output is constructed by the following methods: Entity conversion: all entity categories of each original text are traversed to obtain entities in each entity category, and each entity is combined into a sentence in the form of "entity + is + entity category", and each entity sentence obtained is spliced by ","; Relation conversion: all relation labels of each original text are traversed to obtain relations in each relation label and subjects and objects corresponding to each relation, and each relation is combined into a sentence in the form of "subject + of + relation + is + object", and each relation sentence obtained is spliced by ","; The converted entity sentence and the relation sentence corresponding to each original text are spliced by "," to obtain the target output corresponding to each original text. 5.The method of claim 3, wherein, The method further comprises auxiliary tasks: entity extraction, entity classification, relation subject and object extraction, and relation classification, wherein: The task description of the entity extraction is to extract all entities in the original text; The task description of the entity classification is to identify the entity categories corresponding to the entities in the original text; The task description of the relation subject and object extraction is to identify the subjects and objects of the relations in the original text; The task description of the relation classification is to identify the relations between the subjects and the objects in the original text. 6.The method of claim 5, wherein, The data in the data set D ALL is divided into 7 parts according to a preset proportion, and corresponding target outputs are constructed according to 7 task descriptions. After construction, the data is combined and shuffled to construct a training sample set. 7.The method of claim 1, wherein, The loss iteration is performed by the following formula: wherein, is the predicted probability for time step t, is the true probability for time step t; The loss of all time steps is added to obtain the total loss Loss T : T is all time steps.

8. A computer device, comprising: The method comprises the following steps: The method comprises the following steps: The memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the training method of the named entity and relation joint recognition model in any one of claims 1-7.

Images