End-to-end entity linking model training method, entity linking method and device

Abstract

The present invention provides an end-to-end entity linking model training method, an entity linking method and a device, constructs an initial entity linking model including a first BERT model, a second BERT model and a Global Pointer layer, performs reference recognition based on the first BERT model and the Global Pointer layer, and performs entity disambiguation based on the first BERT model and the second BERT model. During the training process of the model, the loss functions of the two parts of reference recognition and entity disambiguation are combined, and at the same time, the two parts of the parameters of the initial entity linking model used to perform reference recognition and entity disambiguation are back-propagated and adjusted, effectively transmitting and adjusting the two parts of the error, making them interdependent, and improving the overall effect. In the process of entity disambiguation, introducing a preset knowledge base as external knowledge can greatly improve the disambiguation effect.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology, and in particular to an end-to-end entity linking model training method, an entity linking method and a device. Background Art

[0002] Due to the diversity of natural language expressions, there are problems of polysemy and synonymy. Entity linking technology is an effective method to solve the problem of information ambiguity. At present, there are two main methods for entity linking: one is the two-stage method of entity recognition and entity disambiguation for entity linking. The other is to use deep neural networks for end-to-end entity linking. The first technology divides entity linking into two independent stages, and the results of the first stage are used as the input of the second stage; the second technology uses a bidirectional LSTM network (bidirectional long short-term memory network) to obtain all possible references, and then calculates the similarity with all candidate entities. The similarity above a certain threshold is the prediction result. However, the above methods have their own problems to a greater or lesser extent. The first method divides the entity linking task into two independent stages, does not use the dependency between the two stages, and also makes the error of the first stage uncorrectable. The ability of the bidirectional LSTM network to express features is limited compared to the pre-trained model, and it is very dependent on the results of previous work. Therefore, a new entity linking method is urgently needed. Summary of the invention

[0003] In view of this, the embodiments of the present invention provide an end-to-end entity linking model training method, an entity linking method and a device to eliminate or improve one or more defects existing in the prior art, and to solve the problem of insufficient accuracy caused by the relatively independent errors in the two stages of entity recognition and entity disambiguation in the prior art that cannot be transmitted and corrected.

[0004] The technical solution of the present invention is as follows:

[0005] On the one hand, the present invention provides an end-to-end entity linking model training method, comprising:

[0006] Obtain a training sample set, wherein the training sample set includes multiple samples, each sample is a paragraph including one sentence or multiple consecutive sentences, and all entities in the sample and corresponding description information are marked as labels;

[0007] Acquire a preset knowledge base, wherein the preset knowledge base records description information of multiple known entities, wherein the known entities include multiple relationships of multiple meanings of a word and multiple synonyms of multiple words;

[0008] An initial entity linking model is obtained, wherein the initial entity linking model includes a first BERT (Bidirectional Encoder Representations from Transformer) model, a second BERT model and a Global Pointer layer; the first BERT model is connected to the Global Pointer layer to perform reference identification and obtain all predicted entities in the sentence or paragraph of each sample; the first BERT model is connected to the second BERT model to perform entity disambiguation on all predicted entities one by one, wherein, for a specified predicted entity, all outputs of the first BERT model are used as a first vector representation, and the part of the first vector representation corresponding to the specified predicted entity is used as an entity vector representation, and the entity vector representation and the first vector representation are weighted and summed to obtain an overall vector representation; according to the specified predicted entity, the preset knowledge base is searched to find a positive example with the same meaning as the specified predicted entity and two negative examples with different meanings, and the second BERT model performs vector representation operations on the description information of the positive example and the negative example corresponding to the specified predicted entity to obtain a corresponding second vector representation; the overall vector representation is respectively connected with the second vector representations of the positive example and the negative example corresponding to the specified predicted entity, and input into a fully connected layer for scoring to determine whether it is synonymous;

[0009] The initial entity linking model is trained using the training sample set, wherein all predicted entities in each sample are entity disambiguated one by one with the positive examples and negative examples in the preset knowledge base corresponding to each predicted entity, a joint loss function is calculated and back-propagated, and the parameters of the first BERT model, the second BERT model and the Global Pointer layer are adjusted as a whole to obtain a target entity linking model.

[0010] In some embodiments, the preset knowledge base configures candidate words for the known entity that have polysemy or multiple synonymy relationships.

[0011] In some embodiments, when the initial entity linking model is trained using the training sample set, the learning rate is set to 2e-5, and the Adam algorithm is used for gradient descent.

[0012] In some embodiments, the calculation formula of the joint loss function LOSS is:

[0013] Loss = λloss md +(1-λ)loss ed ;

[0014] Among them, loss mdis the loss of the initial entity linking model for reference recognition, loss ed is the loss of entity disambiguation performed by the initial entity linking model, λ is the weight coefficient, 0<λ<1.

[0015] In some embodiments, loss md And loss ed The cross entropy loss function is used and λ is 0.1.

[0016] In some embodiments, the entity vector representation and the first vector representation are weighted and summed to obtain an overall vector representation, the weight ratio of the entity vector representation is 0.7, and the weight ratio of the first vector representation is 0.3.

[0017] On the other hand, the present invention also provides an end-to-end entity linking method, comprising:

[0018] Obtaining a corpus to be processed and a preset knowledge base, wherein the preset knowledge base records description information of multiple known entities, wherein the known entities include multiple relationships of multiple meanings of a word and multiple synonyms of multiple words;

[0019] The corpus to be processed is input into the target entity linking model obtained in the above-mentioned end-to-end entity linking model training method, and the first BERT model and the Global Pointer layer connected in sequence in the target entity linking model perform reference recognition, and the preset knowledge base is queried one by one for the recognized entity evidence to obtain multiple candidate words with polysemous relationships for each entity, and the first BERT model and the second BERT model in the target entity linking model perform entity disambiguation on all entities one by one, wherein, for a specified entity, all the outputs of the first BERT model are used as the first vector representation, and the part of the first vector representation corresponding to the specified entity is used as the entity vector representation, and the entity vector representation and the first vector representation are weighted and summed to obtain the overall vector representation; the second BERT model respectively embeds the description information of the multiple candidate words corresponding to the specified entity to obtain the corresponding second vector representation, and the overall vector representation is respectively connected with the second vector representation of the candidate words corresponding to the specified entity, and input into the fully connected layer for scoring to determine whether it is synonymous to eliminate ambiguity;

[0020] Output multiple entities in the corpus to be processed obtained by identifying the target entity linking model and description information corresponding to each entity in the preset knowledge base.

[0021] In some embodiments, the preset knowledge base encodes and marks multiple description information containing polysemy in each known entity, and encodes the corresponding description information for the entity output by the target entity linking model.

[0022] On the other hand, the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the steps of the above method.

[0023] On the other hand, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor.

[0024] The beneficial effects of the present invention are at least:

[0025] In the end-to-end entity linking model training method, entity linking method and device described in the present invention, an initial entity linking model including a first BERT model, a second BERT model and a Global Pointer layer is constructed, reference recognition is performed based on the first BERT model and the Global Pointer layer, and entity disambiguation is performed based on the first BERT model and the second BERT model. During the training process of the model, the loss functions of the two parts of reference recognition and entity disambiguation are combined, and at the same time, the two parts of the parameters of the initial entity linking model used to perform reference recognition and entity disambiguation are back-propagated and adjusted, effectively transferring and adjusting the two parts of the error, making them interdependent, and improving the overall effect. In the process of entity disambiguation, introducing a preset knowledge base as external knowledge can greatly improve the disambiguation effect.

[0026] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically pointed out in the written description, claims, and drawings.

[0027] Those skilled in the art will appreciate that the objectives and advantages that can be achieved by the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of the present application, and do not constitute a limitation of the present invention. In the drawings:

[0029] Figure 1 A schematic diagram of the structure of an initial entity linking model in an end-to-end entity linking model training method according to an embodiment of the present invention;

[0030] Figure 2A schematic diagram of an initial entity linking model training process in an end-to-end entity linking model training method according to another embodiment of the present invention;

[0031] Figure 3 A schematic diagram of an initial entity linking model testing process in an end-to-end entity linking model training method according to another embodiment of the present invention;

[0032] Figure 4 The present invention is a schematic diagram of the initial entity linking model testing process in the end-to-end entity linking model training method described in another embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0034] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0035] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0036] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0037] It should be noted in advance that entity linking is to map certain strings in a text to the corresponding entities in the knowledge base. In many cases, there are entities with the same name or entities with different names, so this mapping process needs to be disambiguated. For example, for the text "I am reading Harry Potter", "Harry Potter" should refer to the entity "Harry Potter (book)" instead of the entity "Harry Potter series of movies". Current entity linking generally recognizes the scope of entity names (generally called mention), and the work that needs to be done is mainly to disambiguate entities (called entities). There are also some works that do entity recognition and entity disambiguation at the same time, which becomes an end-to-end task.

[0038] In the prior art, most entity linking models adopt an independent two-stage method, first performing reference recognition and then performing the disambiguation. This will lead to the loss of dependency between the two stages and will cause irreversible error transmission problems. The error in the reference recognition stage will be transmitted to the entity disambiguation stage and cannot be adjusted. The present invention performs end-to-end Chinese entity linking based on a pre-trained model. By designing an end-to-end pre-training model, the two stages of reference recognition and entity disambiguation are jointly trained, making full use of the connection and dependency between the two stages, while optimizing the results of the two stages, so that the information of the two stages is effectively integrated with each other. Experiments have shown that by using an end-to-end method, the effects of reference recognition and entity linking are improved.

[0039] Specifically, the present invention provides an end-to-end entity linking model training method, referring to Figure 1 , including steps S101 to S104:

[0040] Step S101: Obtain a training sample set, which includes multiple samples. Each sample is a paragraph containing one sentence or multiple consecutive sentences. All entities in the sample and corresponding description information are marked as labels.

[0041] Step S102: obtaining a preset knowledge base, in which description information of a plurality of known entities is recorded. The known entities include a plurality of relationships between a word with multiple meanings and multiple synonyms.

[0042] Step S103: obtaining an initial entity linking model, which includes a first BERT model, a second BERT model and a Global Pointer layer; the first BERT model is connected to the Global Pointer layer to perform reference identification and obtain all predicted entities in sentences or paragraphs in each sample; the first BERT model is connected to the second BERT model to perform entity disambiguation on all predicted entities one by one, wherein, for a specified predicted entity, all outputs of the first BERT model are used as a first vector representation, the part of the first vector representation corresponding to the specified predicted entity is used as an entity vector representation, and the entity vector representation and the first vector representation are weighted and summed to obtain an overall vector representation; according to the specified predicted entity, a preset knowledge base is searched to find a positive example with the same meaning as the specified predicted entity and two negative examples with different meanings, and the second BERT model performs vector representation operations on the description information of the positive example and the negative example corresponding to the specified predicted entity to obtain a corresponding second vector representation; the overall vector representation is connected to the second vector representations of the positive example and the negative example corresponding to the specified predicted entity, and input into the fully connected layer for scoring to determine whether it is synonymous.

[0043] Step S104: Use the training sample set to train the initial entity linking model, wherein all predicted entities in each sample are disambiguated one by one with the positive and negative examples in the preset knowledge base corresponding to each predicted entity, the joint loss function is calculated and back-propagated, and the parameters of the first BERT model, the second BERT model and the Global Pointer layer are adjusted as a whole to obtain the target entity linking model.

[0044] In step S101, in the training sample set, each sample is a manually annotated corpus. In some embodiments, the database can be directly constructed using existing corpus, or the corpus of the target category can be re-annotated to construct the sample. Specifically, the named entities in each sample are annotated, and the annotation should at least include the location and description information of each entity. Furthermore, the description information can be expressed according to preset rules, or it can be directly linked to the corresponding known entity in the preset knowledge base and marked by coding.

[0045] In step S102, the preset knowledge base is used to record the structured information of multiple corpora, and corresponding description information is configured for the known entities in each corpus. Furthermore, there are polysemy and synonymy relationships between the multiple known entities in the preset knowledge base. In some embodiments, the preset knowledge base configures candidate words with polysemy or synonymy relationships for the known entities. For each known entity, the words with polysemy and synonymy relationships are used as candidate words to construct a query subset to improve the retrieval speed. In addition, multiple description information of the same known entity and the same words with multiple meanings are encoded and marked separately. The coding marking of each entity in the sample in step S101 should be consistent with the marking of the corresponding known entity in the preset knowledge base.

[0046] In step S103, this embodiment constructs an initial entity linking model that integrates reference recognition and entity disambiguation. Figure 1The structure shown. The initial entity linking model includes the first BERT model, the second BERT model and the Global Pointer layer; the full name of the BERT model is: Bidirectional Encoder Representations from Transformer. The goal of the BERT model is to use large-scale unlabeled corpus training to obtain the semantic representation of text containing rich semantic information. The Global Pointer layer uses the idea of ​​global normalization to perform named entity recognition (NER), which can identify nested entities and non-nested entities indiscriminately. Accordingly, the input form of the corpus in the sample to the first BERT model or the second BERT model and the output form are the input and output forms of the general BERT model.

[0047] Specifically, refer to Figure 2 In this embodiment, the first BERT model is connected to the Global Pointer layer to perform the reference recognition task, and the first BERT model is shared in the reference recognition task and the entity disambiguation task. During the training process, after the reference recognition part of the initial entity linking model obtains the predicted entity, it searches the predicted entity one by one in the preset knowledge base, and obtains one positive example and two negative examples corresponding to each predicted entity. In the actual training process, the entity in each sample is marked with the encoding of its corresponding known entity and corresponding meaning description information in the preset knowledge base. The known entity that is identical to the predicted entity word and has the same meaning is a positive example, and the known entity that is identical to the predicted entity word but has a different meaning is a negative example.

[0048] During the entity disambiguation training process, one positive example and two negative examples are selected for each predicted entity in each sample and trained separately, so the predicted entity of one sample corresponds to three sets of data for training.

[0049] In some embodiments, the entity vector representation and the first vector representation are weighted and summed to obtain the overall vector representation, the weight ratio of the entity vector representation is 0.7, and the weight ratio of the first vector representation is 0.3.

[0050] In step S104, the aforementioned initial entity linking model is trained using a training sample set, and in each training iteration, a joint loss is constructed using the losses of the reference recognition and entity disambiguation parts in the initial entity linking model, and back-propagated, while adjusting the model parameters of the reference recognition and entity disambiguation parts.

[0051] In some embodiments, the calculation formula of the joint loss function LOSS is:

[0052] Loss = λloss md +(1-λ)lossed ; (1)

[0053] Among them, loss md is the loss of the initial entity linking model for reference recognition, loss ed is the loss of entity disambiguation performed by the initial entity linking model, λ is the weight coefficient, 0<λ<1. In some embodiments, loss md And loss ed The cross entropy loss function is used and λ is 0.1.

[0054] In some embodiments, when the initial entity linking model is trained using a training sample set, the learning rate is set to 2e-5, and the Adam algorithm is used for gradient descent.

[0055] By using the model proposed in this embodiment, the error transmission and dependency loss problems existing in the two stages can be effectively solved. The reference recognition model and the entity disambiguation model are trained together, and the error in the reference recognition stage can be optimized during the training process, and will not be further transmitted to the entity disambiguation stage to reduce the overall effect. By utilizing the dependency relationship between the two stages, the candidate entity descriptions and knowledge base information of the entity disambiguation model introduce external knowledge to the reference recognition model, which greatly improves the effect of the reference recognition stage. By using this model, the F1 values ​​of both reference recognition and entity disambiguation are improved.

[0056] On the other hand, the present invention also provides an end-to-end entity linking method, comprising steps S201 to S203:

[0057] Step S201: obtaining a corpus to be processed and a preset knowledge base, wherein the preset knowledge base records description information of a plurality of known entities, and the known entities include a plurality of relationships between a word with multiple meanings and multiple synonyms.

[0058] Step S202: Input the corpus to be processed into the target entity linking model obtained in the end-to-end entity linking model training method described in the above steps S101 to S104, and perform reference recognition by the first BERT model and the Global Pointer layer connected in sequence in the target entity linking model, query the preset knowledge base one by one for the recognized entity evidence, obtain multiple candidate words with polysemy for each entity, and perform entity disambiguation on all entities one by one by the first BERT model and the second BERT model in the target entity linking model, wherein, for a specified entity, all the outputs of the first BERT model are used as the first vector representation, the part of the first vector representation corresponding to the specified entity is used as the entity vector representation, and the entity vector representation and the first vector representation are weighted and summed to obtain the overall vector representation; the second BERT model respectively embeds the description information of the multiple candidate words corresponding to the specified entity to obtain the corresponding second vector representation, and connects the overall vector representation with the second vector representation of the candidate words corresponding to the specified entity, and inputs it into the fully connected layer for scoring to determine whether it is synonymous to eliminate ambiguity.

[0059] Step S203: Output multiple entities in the corpus to be processed obtained by identifying the target entity linking model and description information corresponding to each entity in the preset knowledge base.

[0060] Specifically, in step S201, the target entity linking model obtained by training based on steps S101 to S104 needs to be used in conjunction with a preset knowledge base in the process of reference identification and entity disambiguation. Therefore, when performing entity linking, the present invention must first obtain the corpus to be processed and the preset knowledge base. The preset knowledge base used here should be consistent with the form used to train the model in step S102 above, and there may be differences in content. Specifically, the preset knowledge base used in step S201 can configure multiple candidate words with polysemy or polysynonymy relationships for each known entity, and mark them to improve retrieval efficiency. In some embodiments, the preset knowledge base encodes and marks the multiple description information containing polysemy in each known entity, and the encoding of the corresponding description information is marked for the entity output by the target entity linking model.

[0061] In step S202, the corpus to be processed is input into the target entity linking model obtained in the end-to-end entity linking model training method described in steps S101 to S104, wherein the part for referential identification first obtains entities in the corpus to be processed, and then disambiguates the identified entities through the entity disambiguation part. In the disambiguation process, the disambiguation is performed with the help of known entities in the preset knowledge base.

[0062] In step S203, the target entity linking model identifies and outputs entities in the corpus to be processed, and outputs description information corresponding to each entity after disambiguation processing. Specifically, the entity linking relationship corresponding to each identified entity in the preset knowledge base is output, which can be marked in the form of coding.

[0063] On the other hand, the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the steps of the above method.

[0064] On the other hand, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor.

[0065] The present invention is described below in conjunction with specific embodiments:

[0066] This embodiment provides an end-to-end entity linking model, and its working process is divided into the following steps:

[0067] 1) Perform reference identification on the input sentence. Since the input is a whole sentence, it is necessary to identify the reference in the sentence first. The method of reference identification is to use BERT+Global Pointer. 2) Perform entity disambiguation on the identified reference. By extracting the vector representation of the input sentence and the vector representation of the reference to be disambiguated, they are spliced ​​with the vector representations of all possible candidate entity descriptions, and the final link result is predicted through a fully connected layer. 3) Model architecture design. This design is the key technical point of this model. The model of the reference identification model and the model of entity disambiguation form a dual-tower structure. The reference identification model is used for reference identification, and then the reference identification model is reused to extract the vector representation of the input sentence to participate in entity disambiguation. In this way, the joint training and mutual optimization of the reference identification model and the entity disambiguation model are achieved during the training process.

[0068] The reference identification stage can be regarded as a named entity recognition process, where a sentence is input and the predicted entities in the text are output. The entities identified in entity linking become references, which are usually ambiguous and need to be linked and eliminated in the subsequent stages. For example, the reference identification model is as follows Figure 3 As shown in the figure, the sentence to be recognized, "Zhao, the actor who plays Liu, comments on the 'big' in The Great Yangko", is input into the text and passed through the BERT model, and then through the Global Pointer layer, and the predicted entities "Liu, Zhao, The Great Yangko" are output. Global Pointer uses the idea of ​​global normalization to perform named entity recognition (NER), which can effectively solve the problem of nested entities in the text that are difficult to recognize.

[0069] In the entity disambiguation phase, such as Figure 3 As shown in the figure, for the reference identified in the reference recognition stage, all references are traversed in turn as objects to be disambiguated, and one positive example and two negative examples are selected from the candidate entity set corresponding to the reference in the knowledge base as samples to participate in the training of the model, and the description text of the selected entity is used as the input of the BERT model on the right.

[0070] The embedding (word embedding, vector representation) corresponding to "Liu" in the BERT model on the left is weighted summed with the embedding of the last layer of the BERT model on the left to obtain the entity embedding, which is then concatenated with the sentence embedding of the BERT model on the right, and a fully connected layer is used to predict whether the link is correct. Among them, the embedding of the BERT model on the left represents the vector representation of the reference on the left, and the embedding of the BERT model on the right represents the vector representation of the candidate entity description. The concatenation of the embeddings corresponding to the reference introduces the information of the reference, and experiments have shown that it effectively improves the effect of the model.

[0071] During the testing phase, Figure 4 As shown in Figure 1, all the references identified by the reference recognition module need to be linked through the disambiguation model. When linking, all candidate entities of the reference are also sampled, and finally the entity with the highest score is selected as the final linked entity. The correctness of entity linking is calculated to obtain the test result.

[0072] In this embodiment, for model training, the dimension of BERT word vector is set to 768, the maximum length of sentence is 256, the learning rate is set to 2e-5, and the weights of mention embedding and sentence embedding in entity disambiguation stage are 0.7 and 0.3. The loss function of both the reference recognition model and the entity disambiguation model adopts the cross entropy loss function, and the joint loss function of the overall model is:

[0073] loss = λloss md +(1-λ)loss ed ; (1)

[0074] Among them, loss md is the loss of the initial entity linking model for reference recognition, loss ed is the loss of entity disambiguation performed by the initial entity linking model, λ is the weight coefficient, 0<λ<1. In some embodiments, loss md And loss ed The cross entropy loss function is used and λ is 0.1.

[0075] The training data used in this embodiment is the dataset of entity linking tasks for Chinese short texts in ccks2019 (2019 National Knowledge Graph and Semantic Computing Conference), and the knowledge base is the knowledge base compiled by Baidu provided in the dataset. The Adam algorithm is used for gradient descent.

[0076] The end-to-end model of this embodiment combines the reference recognition model and the entity disambiguation model, and trains the reference recognition model and the entity disambiguation model at the same time in the training stage, making full use of the dependency between the two stages. The model has the following advantages: the two models are jointly trained, and the error of the reference recognition model can be optimized and corrected during the training process, while the error of reference recognition in the two independent stages is uncorrectably transferred to the entity disambiguation stage. The disambiguation model introduces external knowledge base information. On the one hand, it can introduce additional information to help the reference recognition model better predict the reference in the text. On the other hand, the introduction of knowledge base information can well optimize the boundary recognition error problem existing in reference recognition. In the end-to-end disambiguation model, a large number of positive and negative samples generated in the reference recognition stage are used to introduce more training data for the entity disambiguation stage. On the one hand, more references can avoid entity omissions, and on the other hand, more data improves the effect and generalization performance of the disambiguation model. Experiments have shown that the end-to-end model has improved the effects of both reference recognition and entity disambiguation, which fully demonstrates the effectiveness of the model.

[0077] In summary, the end-to-end entity linking model training method, entity linking method and device construct an initial entity linking model including a first BERT model, a second BERT model and a Global Pointer layer, perform reference recognition based on the first BERT model and the Global Pointer layer, and perform entity disambiguation based on the first BERT model and the second BERT model. During the training of the model, the loss functions of the two parts of reference recognition and entity disambiguation are combined, and at the same time, the two parts of the parameters of the initial entity linking model used to perform reference recognition and entity disambiguation are back-propagated and adjusted, effectively transferring and adjusting the two parts of the error, making them interdependent, and improving the overall effect. In the process of entity disambiguation, introducing a preset knowledge base as external knowledge can greatly improve the disambiguation effect.

[0078] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium or transmitted on a transmission medium or a communication link via a data signal carried in a carrier. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memory, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, and the like. Code segments can be downloaded via computer networks such as the Internet, intranets, and the like.

[0079] It should also be noted that the exemplary embodiments mentioned in the present invention describe some methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiments, or in a different order from the embodiments, or several steps can be performed simultaneously.

[0080] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0081] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An end-to-end entity linking model training method, characterized in that: include: Obtain a training sample set, wherein the training sample set includes multiple samples, each sample is a paragraph including one sentence or multiple consecutive sentences, and all entities in the sample and corresponding description information are marked as labels; Acquire a preset knowledge base, wherein the preset knowledge base records description information of multiple known entities, wherein the known entities include multiple relationships of multiple meanings of a word and multiple synonyms of multiple words; An initial entity linking model is obtained, wherein the initial entity linking model includes a first BERT model, a second BERT model and a Global Pointer layer; the first BERT model is connected to the Global Pointer layer to perform reference identification and obtain all predicted entities in the sentence or paragraph of each sample; the first BERT model is connected to the second BERT model to perform entity disambiguation on all predicted entities one by one, wherein, for a specified predicted entity, all outputs of the first BERT model are used as a first vector representation, the part of the first vector representation corresponding to the specified predicted entity is used as an entity vector representation, and the entity vector representation and the first vector representation are weighted and summed to obtain an overall vector representation; according to the specified predicted entity, the preset knowledge base is searched to find a positive example with the same meaning as the specified predicted entity and two negative examples with different meanings, and the second BERT model performs vector representation operations on the description information of the positive example and the negative example corresponding to the specified predicted entity to obtain a corresponding second vector representation; the overall vector representation is respectively connected with the second vector representations of the positive example and the negative example corresponding to the specified predicted entity, and input into a fully connected layer for scoring to determine whether it is synonymous; The initial entity linking model is trained using the training sample set, wherein all predicted entities in each sample are entity disambiguated one by one with the positive examples and negative examples in the preset knowledge base corresponding to each predicted entity, a joint loss function is calculated and back-propagated, and the parameters of the first BERT model, the second BERT model and the GlobalPointer layer are adjusted as a whole to obtain a target entity linking model.

2. The end-to-end entity linking model training method according to claim 1, characterized in that: The preset knowledge base configures candidate words with polysemy or synonymy for the known entity.

3. The end-to-end entity linking model training method according to claim 1, characterized in that: When the initial entity linking model is trained using the training sample set, the learning rate is set to 2e-5, and the Adam algorithm is used for gradient descent.

4. The end-to-end entity linking model training method according to claim 1, characterized in that: The calculation formula of the joint loss function LOSS is: Loss=λloss md +(1-λ)loss ed ; Among them, loss md is the loss of the initial entity linking model for reference recognition, loss ed is the loss of entity disambiguation performed by the initial entity linking model, λ is the weight coefficient, 0<λ<1.

5. The end-to-end entity linking model training method according to claim 4, characterized in that: loss md And loss ed The cross entropy loss function is used and λ is 0.

1.

6. The end-to-end entity linking model training method according to claim 1, characterized in that: The entity vector representation and the first vector representation are weightedly summed to obtain an overall vector representation, where the weight ratio of the entity vector representation is 0.7, and the weight ratio of the first vector representation is 0.

3.

7. An end-to-end entity linking method, characterized in that: include: Obtaining a corpus to be processed and a preset knowledge base, wherein the preset knowledge base records description information of multiple known entities, wherein the known entities include multiple relationships of multiple meanings of a word and multiple synonyms of multiple words; The corpus to be processed is input into the target entity linking model obtained in the end-to-end entity linking model training method as described in any one of claims 1 to 6, and the first BERT model and the Global Pointer layer connected in sequence in the target entity linking model perform reference recognition, query the preset knowledge base one by one for the recognized entity evidence, and obtain multiple candidate words with polysemous relationships for each entity, and the first BERT model and the second BERT model in the target entity linking model perform entity disambiguation on all entities one by one, wherein, for a specified entity, all the outputs of the first BERT model are used as the first vector representation, the part of the first vector representation corresponding to the specified entity is used as the entity vector representation, and the entity vector representation and the first vector representation are weighted and summed to obtain the overall vector representation; the second BERT model respectively embeds the description information of the multiple candidate words corresponding to the specified entity to obtain the corresponding second vector representation, and the overall vector representation is respectively connected with the second vector representation of the candidate words corresponding to the specified entity, and input into the fully connected layer for scoring to determine whether they are synonymous to eliminate ambiguity; Output multiple entities in the corpus to be processed obtained by identifying the target entity linking model and description information corresponding to each entity in the preset knowledge base.

8. The end-to-end entity linking method according to claim 7, characterized in that: The preset knowledge base encodes and marks multiple description information of each known entity containing a polysemous word, and encodes the corresponding description information of the entity output by the target entity linking model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images