Entity prediction method and apparatus

Abstract

The embodiment of the application discloses an entity prediction method and device, wherein the method comprises: inputting a text sequence corresponding to a to-be-predicted triple into a first entity representation model, wherein the to-be-predicted triple contains a known entity, a relationship between the known entity and a to-be-predicted entity, and the to-be-predicted entity, and obtaining a representation vector of the to-be-predicted entity; querying the representation vector of the to-be-predicted entity in a pre-constructed knowledge repository to obtain an entity that satisfies a preset similarity requirement between the representation vector and the representation vector of the to-be-predicted entity as a candidate entity, wherein the knowledge repository contains representation vectors corresponding to each entity; inputting the representation vector of the to-be-predicted entity and the candidate entity into a first representation mapping model to obtain a prediction confidence of the to-be-predicted entity on each candidate entity; and determining a prediction result of the to-be-predicted entity from the candidate entity according to the prediction confidence of the to-be-predicted entity on each candidate entity. Through the application, entity prediction can be efficiently realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and big data, in particular to an entity prediction method and device. BACKGROUND

[0002] A knowledge graph provides a better way to organize, manage and utilize massive information, and describes entities such as concepts and objects in the real world and their relationships. A knowledge graph is composed of nodes and edges, each node representing an entity existing in the real world, and each edge representing the relationship between entities. At present, knowledge graphs have been widely used in various application scenarios such as improving search quality, question and answer platforms, and risk identification.

[0003] Since a knowledge graph is inherently incomplete, most existing knowledge graphs are sparse, which leads to the introduction of KGC (knowledge graph completion) to add new triples to the knowledge graph. Among them, head entity prediction and tail entity prediction are two kinds of entity prediction for triples, for example, triple (h, r, t) represents predicting the head entity based on the known relationship and tail entity, and triple (h, r,?) represents predicting the tail entity based on the known head entity and relationship. There is an urgent need for an entity prediction method to efficiently implement entity prediction. SUMMARY

[0004] Therefore, the present application provides an entity prediction method and device to efficiently implement entity prediction.

[0005] The present application provides the following solutions:

[0006] In a first aspect, an entity prediction method is provided, which comprises:

[0007] inputting a text sequence corresponding to a to-be-predicted triple into a first entity representation model, wherein the to-be-predicted triple comprises a known entity, a relationship between the known entity and a to-be-predicted entity, and the to-be-predicted entity, to obtain a representation vector of the to-be-predicted entity;

[0008] querying the representation vector of the to-be-predicted entity in a pre-constructed knowledge repository to obtain an entity whose similarity between the representation vector and the representation vector of the to-be-predicted entity meets a preset similarity requirement as a candidate entity, wherein the knowledge repository comprises representation vectors corresponding to each entity;

[0009] inputting the representation vector of the to-be-predicted entity and the candidate entity into a first representation mapping model to obtain a prediction confidence of the to-be-predicted entity on each candidate entity;

[0010] According to a prediction confidence of the to-be-predicted entity on each candidate entity, a prediction result of the to-be-predicted entity is determined from the candidate entities.

[0011] According to an implementable manner in embodiments of the present application, the first entity representation model and the first representation mapping model are pre-trained in the following manner:

[0012] Obtaining first training data, the first training data comprising a plurality of triple samples, the triple samples comprising two entities and a relationship between the two entities;

[0013] Training the first entity representation model and the first representation mapping model using the first training data; wherein a first text sequence corresponding to the triple sample is input into the first entity representation model, one entity in the first text sequence corresponding to the triple sample is masked, the first entity representation model outputs a representation vector corresponding to the masked entity to the first representation mapping model, and the first representation mapping model outputs a prediction result of the masked entity using the representation vector corresponding to the masked entity; the training target is to minimize the difference between the prediction result output by the first representation mapping model and the masked entity in the triple sample, and the first entity representation model uses a pre-trained language model.

[0014] According to an implementable manner in embodiments of the present application, the knowledge repository is pre-built in the following manner:

[0015] For each entity in the entity library, the following are performed respectively: obtaining a triple comprising the entity, masking the entity in a first text sequence corresponding to the triple, and inputting the first text sequence corresponding to the triple into the first entity representation model to obtain a representation vector of the entity and store the representation vector into the knowledge repository.

[0016] According to an implementable manner in embodiments of the present application, the knowledge repository is pre-built in the following manner:

[0017] For each entity in the entity library or each entity extracted from the page, the following are performed respectively: learning a representation vector of the entity from a description text comprising the entity and the entity using a pre-trained language model based on a prompt learning mechanism, and storing the representation vector of the entity into the knowledge repository.

[0018] According to an implementable manner in embodiments of the present application, the learning of the representation vector of the entity from the description text comprising the entity and the entity using the pre-trained language model based on the prompt mechanism comprises:

[0019] Obtaining second training data, the second training data comprising an entity and a description text of the entity;

[0020] training a second entity representation model and a second representation mapping model using the second training data, wherein an entity in the second training data is masked and a description text of the entity is filled into a preset prompt template to obtain a second text sequence, the second text sequence is input into the second entity representation model, the second entity representation model outputs a representation vector corresponding to the masked entity to the second representation mapping model, and the second representation mapping model outputs a prediction result of the masked entity using the representation vector corresponding to the masked entity; a training target is to minimize a difference between the prediction result output by the second representation mapping model and the masked entity; and the second entity representation model uses a pre-trained language model;

[0021] After the training is completed, a representation vector of each entity in the second training data is obtained from the second entity representation model and is stored in the knowledge base.

[0022] According to an implementable manner in the embodiments of the present application, the prompt template includes a slot of the masked entity, a slot of the description text of the entity, and prompt information.

[0023] The prompt information is used to semantically prompt a content type that needs to be predicted by the masked part.

[0024] According to an implementable manner in the embodiments of the present application, the prediction result of the to-be-predicted entity is determined from the candidate entities according to a prediction confidence of the to-be-predicted entity on each candidate entity, including:

[0025] For each candidate entity, a similarity between a representation vector of the candidate entity and the representation vector of the to-be-predicted entity and the prediction confidence of the to-be-predicted entity on the candidate entity are weighted to obtain a score of the candidate entity.

[0026] The candidate entity with the highest score is selected from the candidate entities as the prediction result of the to-be-predicted entity.

[0027] In a second aspect, an entity prediction apparatus is provided, and the apparatus includes:

[0028] a representation obtaining unit configured to input a text sequence corresponding to a to-be-predicted triple into a first entity representation model, wherein the to-be-predicted triple includes a known entity, a relationship between the known entity and a to-be-predicted entity, and the to-be-predicted entity, to obtain a representation vector of the to-be-predicted entity;

[0029] a candidate query unit configured to query a pre-constructed knowledge base using the representation vector of the to-be-predicted entity to obtain entities whose similarity between the representation vector and the representation vector of the to-be-predicted entity meets a preset similarity requirement as candidate entities, wherein the knowledge base includes representation vectors corresponding to entities.

[0030] a representation mapping unit, configured to input a representation vector of the to-be-predicted entity and the candidate entities into a first representation mapping model to obtain a prediction confidence of the to-be-predicted entity on each candidate entity;

[0031] a result determination unit, configured to determine a prediction result of the to-be-predicted entity from the candidate entities according to the prediction confidence of the to-be-predicted entity on each candidate entity.

[0032] According to a third aspect, a computer readable storage medium is provided, having stored thereon a computer program which, when executed by a processor, implements the steps of the method of any one of the above first aspect.

[0033] According to a fourth aspect, an electronic device is provided, comprising:

[0034] one or more processors; and

[0035] a memory associated with the one or more processors, the memory configured to store program instructions, the program instructions, when read and executed by the one or more processors, performing the steps of the method of any one of the above first aspect.

[0036] According to the embodiments provided in the present application, the following technical effects are disclosed:

[0037] 1) According to the predicted representation vector of the to-be-predicted entity, the present application first matches to the candidate entities with similar degrees of proximity in the knowledge repository, and then only needs to map the representation vector of the to-be-predicted entity to the candidate entities to obtain the prediction confidence of the to-be-predicted entity on each candidate entity, and then determines the prediction result of the to-be-predicted entity from the candidate entities according to the prediction confidence. Obviously, this way does not need to traverse all entities in the entity space to determine the prediction confidence of the representation vector of the to-be-predicted entity on all entities, greatly improving the prediction efficiency and enabling efficient knowledge graph completion.

[0038] 2) In the present application, the construction of the knowledge repository is one aspect through learning of the triple containing the entity to obtain the structural representation of the entity in the knowledge graph, and the other aspect is through learning of the text description of the entity to obtain the text representation of the entity, so that the representation vectors of the entities in the knowledge repository are more rich and comprehensive.

[0039] 3) In the present application, for each entity in the entity library or each entity extracted from the page, the following is performed: learning the representation vector of the entity from the entity and the description text of the entity based on the prompt prompt learning mechanism using the pre-trained language model, which can improve the coverage of rare, emerging and never-appeared entities in the knowledge warehouse, timely include them in the knowledge warehouse, enrich the representation vectors of each entity in the knowledge warehouse, and further improve the prediction coverage of entities that appear less or have never appeared in the knowledge graph; on the other hand, it can also make more full use of the knowledge of the pre-trained language model.

[0040] 4) When performing entity prediction, on the one hand, the prediction confidence of the entity to be predicted on the candidate entity is determined, and on the other hand, the similarity of the representation vectors between the entity to be predicted and the candidate entity is determined, and the two factors are comprehensively considered to make the prediction result of the entity more accurate.

[0041] Of course, implementing any product of the present application does not necessarily require all the advantages described above. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0043] Figure 1 An exemplary system architecture diagram to which embodiments of the present application can be applied is shown;

[0044] Figure 2 A flowchart of an entity prediction method provided by an embodiment of the present application;

[0045] Figure 3 A training principle diagram of a first entity representation model and a first representation mapping model provided by an embodiment of the present application;

[0046] Figure 4 A principle diagram of constructing a knowledge warehouse provided by an embodiment of the present application;

[0047] Figure 5 An example diagram of an entity prediction method provided by an embodiment of the present application;

[0048] Figure 6 A schematic block diagram of an entity prediction device provided by an embodiment of the present application;

[0049] Figure 7 A schematic architecture diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application belong to the scope of protection of the present application.

[0051] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0052] It should be understood that the term "and / or" used herein is only to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " herein generally represents an "or" relationship between the front and rear associated objects.

[0053] Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting". Similarly, depending on the context, the phrase "if determined" or "if detecting (a stated condition or event)" can be interpreted as "when determined" or "in response to determining" or "when detecting (a stated condition or event)" or "in response to detecting (a stated condition or event)".

[0054] With the development of large-scale language models, pre-training language models such as BERT (Bidirectional Encoder Representation from Transformers) show excellent effects on natural language processing tasks. In the knowledge graph completion task, BERT is gradually used to solve the entity prediction task in knowledge graph completion. The main idea is to use a BERT-based entity prediction model. After inputting the predicted triple into the entity prediction model, the entity prediction model uses the binary classification task of the predicted triple to calculate the confidence of each entity in the entity space, so as to determine the entity with the highest confidence as the prediction result. However, the main defects of this method include:

[0055] 1) Low running efficiency, which needs to traverse all entities in the entity space, and the complexity is difficult to bear;

[0056] 2) It is difficult to reason about rare, emerging and never-before-seen entities;

[0057] 3) cannot fully utilize and mine the knowledge in the pre-trained language model.

[0058] Therefore, a new entity prediction method is proposed in the present application. In order to facilitate the understanding of the present application, the system architecture to which the present application is applied is first described briefly.

[0059] Figure 1 An exemplary system architecture diagram to which the embodiments of the present application can be applied is shown as shown in FIG. 1. Figure 1 The system mainly includes a knowledge extraction device, a knowledge processing device, a knowledge completion device and a database storing a knowledge graph.

[0060] The knowledge extraction device is responsible for extracting available knowledge units such as entity recognition, relationship extraction, attribute extraction, etc. from unstructured or semi-structured data.

[0061] The knowledge processing unit is responsible for processing the available knowledge units extracted by the knowledge extraction device into elements in the knowledge graph. The knowledge processing involved mainly includes: knowledge fusion, quality evaluation, knowledge update, etc. The knowledge fusion refers to fusing multi-source knowledge to determine equivalent entities, equivalent relationships and equivalent attributes in the knowledge graph, mainly including entity alignment and entity disambiguation processing. The quality evaluation is mainly to evaluate the confidence of the knowledge to ensure the quality of the knowledge graph. The knowledge update is to update the content of the knowledge graph over time or with the increase of new knowledge, to ensure the timeliness of the knowledge.

[0062] The knowledge completion device is responsible for mining or deducing unknown or implicit semantic relationships for the incompleteness of the existing entities or relationships in the knowledge graph, thereby completing the knowledge graph. Knowledge completion mainly includes entity completion, relationship completion, and completion of the structure of the knowledge graph, and the present application mainly relates to entity completion.

[0063] The above-mentioned knowledge extraction device, knowledge processing device and knowledge completion device can be a single server, can be set in the same server, can be a server group composed of multiple servers, or can be a cloud server. The cloud server is also called a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of large management difficulty and weak service scalability in traditional physical host and virtual private server (VP, Virtual Private Server) service.

[0064] It should be understood that Figure 1The number of knowledge extraction devices, knowledge processing devices, knowledge completion devices, and databases storing knowledge graphs in the above-mentioned embodiments is merely illustrative. According to the implementation needs, there can be any number of knowledge extraction devices, knowledge processing devices, knowledge completion devices, and databases storing knowledge graphs.

[0065] Figure 2 The flowchart of the entity prediction method provided by the embodiments of the present application can be executed by the knowledge completion device in the architecture shown in Figure 1 As shown in the architecture shown in Figure 2 The method can include the following steps:

[0066] Step 202: inputting a text sequence corresponding to a to-be-predicted triple into a first entity representation model, wherein the to-be-predicted triple contains a known entity, a relationship between the known entity and a to-be-predicted entity, and the to-be-predicted entity, to obtain a representation vector of the to-be-predicted entity.

[0067] Step 204: querying the representation vector of the to-be-predicted entity in a pre-constructed knowledge repository to obtain an entity that satisfies a preset similarity requirement between the representation vector and the representation vector of the to-be-predicted entity as a candidate entity, wherein the knowledge repository contains representation vectors corresponding to each entity.

[0068] Step 206: inputting the representation vector of the to-be-predicted entity and the candidate entity into a first representation mapping model to obtain a prediction confidence of the to-be-predicted entity in each candidate entity.

[0069] Step 208: determining a prediction result of the to-be-predicted entity from the candidate entities according to the prediction confidence of the to-be-predicted entity in each candidate entity.

[0070] As can be seen from the above flow, in the embodiments of the present application, the representation vector of the to-be-predicted entity is first matched to a candidate entity with similar degree in the knowledge repository, and then only needs to be mapped to the candidate entity to obtain the prediction confidence of the to-be-predicted entity in each candidate entity, and then determine the prediction result of the to-be-predicted entity from the candidate entities according to the prediction confidence. Obviously, this way does not need to traverse all entities in the entity space to determine the prediction confidence of the representation vector of the to-be-predicted entity in all entities, which greatly improves the prediction efficiency and can efficiently implement knowledge graph completion.

[0071] It should be noted that the "first", "second" and the like involved in the embodiments of the present application do not have the limitation of size, order and quantity, and are only used to distinguish in name, for example, "first entity representation model" and "second entity representation model" are used to distinguish two entity representation models in name, "first representation mapping model" and "second representation mapping model" are used to distinguish two representation mapping models in name, and "first training data" and "second training data" are used to distinguish two sets of training data in name.

[0072] The steps in the above process will be described below. First, the step 202, i.e., "inputting the text sequence corresponding to the to-be-predicted triple into the first entity representation model to obtain the representation vector of the to-be-predicted entity", will be described in detail in combination with an embodiment.

[0073] In the embodiments of the present application, the to-be-predicted triple contains a known entity, a relationship between the known entity and the to-be-predicted entity, and the to-be-predicted entity. It mainly includes the following two cases:

[0074] The to-be-predicted triple (h, r,?) is the known head entity, the relationship between the head entity and the tail entity, and the to-be-predicted tail entity.

[0075] The to-be-predicted triple (?, r, t) is the known head entity and the relationship between the head entity and the tail entity, the to-be-predicted tail entity.

[0076] The above to-be-predicted triple can be obtained from the knowledge graph that has been generated, and the missing head entity is obtained, and the known tail entity and the relationship constitute the to-be-predicted triple; or the missing tail entity is obtained, and the known head entity and the relationship constitute the to-be-predicted triple. It can also be obtained from Figure 1 The knowledge processing device in the architecture shown in the figure, for example, after the knowledge processing device obtains entities, relationships and other elements, the missing entity is used to form the to-be-predicted triple. In addition, the to-be-predicted triple can also be obtained from other channels, which will not be listed here.

[0077] In this step, the to-be-predicted triple is spliced in a certain order, in which the known entity, the relationship and the to-be-predicted entity are respectively taken as an element (Token), the known entity, the relationship and the to-be-predicted entity can be inserted with a separator such as "[SEP]", the start position of the text sequence is inserted with a start symbol "[CLS]", and "[SEP]" and "[CLS]" are also taken as a Token. The to-be-predicted entity can use a to-be-predicted identifier to represent that the element is to-be-predicted content.

[0078] For example, assuming that the known head entity is "Xenophon" (a person's name), the relationship is "resides", and the to-be-predicted tail entity, the to-be-predicted triple can be spliced to obtain the following text sequence:

[0079] [CLS] [X] [SEP] [Y] [SEP] [MASK] [SEP]

[0080] Wherein, [MASK] represents a masked element, which can be used to indicate that the element is a to-be-predicted content, i.e., a to-be-predicted entity in the above text sequence. In actual scenarios, other forms can be designed to represent the to-be-predicted content, or additional identifiers can be added to represent the to-be-predicted content, etc. [CLS] is a start symbol, and [SEP] is a separator.

[0081] The first entity representation model involved in this step can encode each element in the text sequence corresponding to the to-be-predicted triple after inputting the text sequence, to obtain a representation vector of the to-be-predicted entity. The first entity representation model can be pre-trained based on a pre-trained language model, and the pre-trained language model can include BERT, RoBerta (Robustly Optimized BERT), GPT (Generative Pre-Training), etc. These pre-trained language models use Transformer as the encoding structure, and after inputting the text sequence corresponding to the to-be-predicted triple, the encoding result of the to-be-predicted entity effectively references the content of the entire text sequence, which can well reflect the structural representation of the to-be-predicted triple.

[0082] The training process of the first entity representation model is described below. As one of the implementable ways, the first entity representation model involved in this step and the first representation mapping model involved in step 206 can be pre-trained by the following process:

[0083] First, the first training data is obtained, which includes a plurality of triple samples, and the triple sample includes two entities and the relationship between the two entities. In this training task, the known triples are used to construct an entity prediction task to train the first entity representation model and the first representation mapping model. For example, a large number of complete triples can be obtained from a knowledge graph, i.e., known head entities, tail entities, and relationships between them, which can be used as samples to construct the first training data.

[0084] Then, the first entity representation model and the first representation mapping model are trained using the first training data. As shown in Figure 3 the first text sequence corresponding to the triple sample is input into the first entity representation model, wherein one of the entities in the first text sequence corresponding to the triple is masked. It should be noted that the selection of the masked entity here is consistent with the prediction type shown in Figure 2 Figure 2 ​If the entity to be predicted in the flow shown is the head entity, then in the training task of the first entity representation model, the entity masked in the triple sample is the head entity. Figure 2 If the entity to be predicted in the flow shown is the tail entity, then in the training task of the first entity representation model, the entity masked in the triple sample is the tail entity.

[0085] For example, the triple sample is (Plato, lives in, Greece), and the first text sequence is obtained by masking "Greece": [CLS] [Plato] [SEP] [lives in] [SEP] [MASK] [SEP]. The first text sequence is input into the first entity representation model. The processing of other triple samples is similar.

[0086] As shown in the middle, the first entity representation model outputs the representation vector of the masked entity to the first representation mapping model, that is, outputs the representation vector of [MASK] to the first representation mapping model. The first representation mapping model outputs the prediction result of the masked entity using the representation vector corresponding to the masked entity, that is, predicts the content corresponding to [MASK]. The first representation mapping model can be a fully connected network, which realizes the mapping of the representation vector to the entity space. Figure 3 The training target of the above training process is to minimize the difference between the prediction result output by the first representation mapping model and the masked entity in the triple sample. Continuing the above example, the training target is to make the entity predicted by the first representation mapping model for [MASK] as consistent as possible with the true value "Greece" of [MASK]. The first representation mapping model predicts the prediction confidence of the masked entity on each entity in the entity space, that is, the probability of the masked entity being each entity in the entity space. The training target is actually to make the probability value of the masked entity on the true value as large as possible.

[0087] In the embodiments of the present specification, a loss function can be constructed according to the above training target. In each iteration, the value of the loss function is used to update the parameters of the first entity representation model and the first representation mapping model by using methods such as gradient descent, until the preset training end condition is met. The training end condition can include, for example, the value of the loss function is less than or equal to the preset loss function threshold, the number of iterations reaches the preset number threshold, and the like.

[0088] After training, the first entity representation model trained is used to predict the representation vector of the entity to be predicted in the triple to be predicted in step 202.

[0089]

[0090] ​The following describes the step 204, i.e., "querying the pre-constructed knowledge repository by using the representation vector of the entity to be predicted to obtain an entity whose similarity between the representation vector and the representation vector of the entity to be predicted meets a preset similarity requirement as a candidate entity", in detail in combination with an embodiment.

[0091] In the embodiment of the present application, the representation vector of each entity in the entity space can be obtained in advance, and each entity and its corresponding representation vector are stored in the knowledge repository. After obtaining the representation vector of the entity to be predicted in step 202, the entity whose similarity between the vector representation and the vector representation of the entity to be predicted meets a preset similarity requirement is queried from the knowledge repository as a candidate entity. The preset similarity requirement can be that the similarity is ranked in the top K, K being a preset positive integer, or the similarity is greater than or equal to a preset similarity threshold. When calculating the similarity between the representation vectors, the L1 distance (i.e., Manhattan distance) between the representation vectors, the L2 distance (i.e., Euclidean distance), etc. can be used.

[0092] As one of the implementable ways, the representation vector of each entity in the entity library can be obtained in advance by the first entity representation model trained above and stored in the knowledge base.

[0093] The first text sequence corresponding to the known triple can be input into the first entity representation model, one entity in the first text sequence is masked as the entity to be predicted, and the representation vector of the masked entity output by the first entity representation model is stored in the knowledge base as the representation vector corresponding to the entity. In this way, the representation vectors of most entities can be obtained, which reflect the structural representation of the entity.

[0094] For example, the known triple (Plato, lives in, Greece) is as shown in part (a) of Figure 4 The text sequence corresponding thereto is "[CLS] [Plato] [SEP] [lives in] [SEP] [MASK] [SEP]", wherein "Greece" is masked, and the representation vector corresponding to the entity "Greece" can be output by the first entity representation model.

[0095] In order to ensure that the entity prediction method of the embodiments of the present application covers rare, emerging and never-before-seen entities, timely incorporate newly emerging entities into the knowledge repository, and enrich the information of each entity representation vector in the knowledge repository, a preferred embodiment is provided to further supplement the above-mentioned acquisition of the representation vector of each entity by the first entity representation model. The idea is to acquire the representation vector of the entity by predicting the task of the entity through the text description of the entity. In order to better utilize the knowledge of the training language model, the Prompt Learning mechanism is adopted in the embodiments of the present application, and the following is performed for each entity in the entity library or each entity extracted from the page: learning the representation vector of the entity from the entity and the description text of the entity by using the pre-trained language model, and storing the representation vector of the entity into the knowledge base. Prompt Learning refers to processing the input text information according to a specific template, and reconstructing the task into a form that can better utilize the pre-trained language model for processing. The specific process is as follows:

[0096] First, the second training data is acquired, and the second training data includes an entity and a description text of the entity. In the embodiments of the present application, the entity and the description text of the entity can be used as samples. The entity can come from an existing entity library or be extracted from a page, so that emerging and newly appearing entities can be timely and effectively incorporated into the knowledge repository. The description text of the entity can be obtained from a knowledge page, a commodity detail page, an official website page, etc.

[0097] For example, a knowledge entry is taken as an entity, and the description text of the entity is extracted from the summary, attributes, first paragraph of the text, etc. in the knowledge page of the knowledge entry. For another example, the name of a commodity can be taken as an entity, and the description text of the commodity is extracted from the introduction, attribute description, etc. of the commodity in the detail page of the commodity. For another example, a person can be taken as an entity, and the description text of the person is extracted from the webpage of the personnel introduction of the website of the organization to which the person belongs. And so on.

[0098] Taking an example, the entry "Greece" is obtained from a knowledge entry, and "Greece is a country in southern Europe and the birthplace of Western civilization" is extracted from the summary of the knowledge webpage of the entry as the description text of the entity "Greece".

[0099] Then, the second entity representation model and the second representation mapping model are trained by using the second training data. As shown in (b) of Figure 4 In (b), the entity in the second training data is masked, and the description text of the entity is filled into the preset prompt template to obtain a second text sequence, and the second text sequence is input into the second entity representation model.

[0100] The prompt learning mechanism is to perform a prompt on a trained model, and after the prompt is completed, the model can directly predict on a specific task without fine-tuning on the downstream task, and only a small amount of labeled data is needed. In the prompt learning mechanism, the input text is obtained by using a pre-constructed prompt template. In the embodiments of the present application, the prompt template includes a masked entity, a slot of a description text corresponding to the entity, and prompt information. The prompt information is used to semantically prompt the content type to be predicted by the masked part.

[0101] The input x of the prompt learning d which can be expressed as:

[0102]

[0103] wherein, represents the description text corresponding to the entity, [MASK] is the masked entity, and prompt() represents a prompt processing of the [MASK] using the prompt information.

[0104] Continuing the above example, the obtained second text sequence can be “[CLS] prompt([MASK])[SEP][Greece is a country in southern Europe, and is the birthplace of Western civilization][SEP]”. The second entity representation model is inputted, wherein [MASK] represents the masked entity, i.e., the content to be predicted. The prompt([MASK]) can be, for example, “the entity is [MASK]”, and the prompt information can also be used to prompt the content type to be predicted by the [MASK] part. Other prompt information is not listed here.

[0105] The second entity representation model outputs the representation vector corresponding to the masked entity to the second representation mapping model, and the second representation mapping model outputs the prediction result of the masked entity using the representation vector corresponding to the masked entity. The first representation mapping model can be a fully connected network, which realizes the mapping of the representation vector to the entity space.

[0106] The training target of the above training process is to minimize the difference between the prediction result output by the second representation mapping model and the corresponding masked entity in the second training data. Continuing the above example, the training target is to make the prediction confidence of the masked entity on each entity in the entity space, i.e., the probability that the masked entity is each entity in the entity space, predicted by the second representation model. The training target is actually to make the probability of the masked entity on the true value as large as possible.

[0107] In the embodiments of the present specification, a loss function can be constructed according to the training target described above, and the parameters of the second entity representation model and the second representation mapping model are updated by using the value of the loss function in each iteration, such as gradient descent, until the preset training end condition is met. The training end condition can include, for example, the value of the loss function is less than or equal to the preset loss function threshold, the number of iterations reaches the preset number threshold, and the like.

[0108] After training, the representation vectors of each entity in the second training data are obtained from the second entity representation model and stored in the knowledge repository. Similarly, the second entity representation model described above is pre-trained based on a pre-trained language model, which can include BERT, RoBerta, GPT, and the like. These pre-trained language models use Transformer as the encoding structure, and after inputting the masked entity and the text sequence corresponding to the description text of the entity, the encoding result of the masked entity effectively references the content of the entire text sequence, thereby effectively expanding the text representation of the masked entity.

[0109] As can be seen from the above embodiments, the representation vector of the entity "Greece" can be obtained by the first entity representation model, which embodies the structural representation of the entity in the triple. The representation vector of the entity "Greece" can also be obtained by learning the entity description text through the prompt mechanism, which embodies the text representation of the entity. That is, there can be multiple representation vectors corresponding to the same entity in the knowledge repository, thereby making the representation vectors of the entities more rich.

[0110] The above step 206, i.e., "inputting the representation vector of the entity to be predicted and the candidate entities into the first representation mapping model to obtain the prediction confidence of the entity to be predicted in each candidate entity", will be described in detail below in conjunction with embodiments.

[0111] The candidate entities obtained in step 204 are obtained by calculating the similarity between the representation vector of the entity to be predicted and the representation vectors of each entity in the knowledge repository, and are entities similar to the entity to be predicted. Therefore, the prediction result of the entity to be predicted is usually one of the candidate entities. Therefore, when predicting the entity to be predicted according to the representation vector of the entity to be predicted, it is not necessary to map the representation vector of the entity to be predicted to each entity in the entity space, but only to the candidate entities to obtain the prediction confidence of the entity to be predicted in the candidate entities. That is, the classification task only needs to be performed on the candidate entities, which greatly reduces the computational complexity and improves the computational efficiency.

[0112] The representation vector of the to-be-predicted entity and the information of the candidate entity are input into the first representation mapping model, and the first representation mapping model outputs the prediction confidence of the to-be-predicted entity in each candidate entity. The process of how to pre-train the first representation mapping model has been described in the previous embodiments, and will not be repeated here.

[0113] The step 208, i.e., “determining the prediction result of the to-be-predicted entity from the candidate entities according to the prediction confidence of the to-be-predicted entity in each candidate entity”, will be described in detail below in combination with an embodiment.

[0114] As one of the implementable modes, only the candidate entity with the highest prediction confidence can be selected as the prediction result of the to-be-predicted entity according to the prediction confidence of the to-be-predicted entity in each candidate entity.

[0115] But as another more preferred embodiment, the prediction result of the to-be-predicted entity can be determined from the candidate entities in combination with the similarity between the representation vector of the candidate entity and the representation vector of the to-be-predicted entity, and the prediction confidence of the to-be-predicted entity in the candidate entity. For example, the similarity between the representation vector of the candidate entity and the representation vector of the to-be-predicted entity and the prediction confidence of the to-be-predicted entity in the candidate entity can be weighted to obtain the score of the candidate entity, and then the candidate entity with the highest score can be selected as the prediction result of the to-be-predicted entity.

[0116] The above method flow will be described by taking a specific example. It is assumed that there is a triple with a missing tail entity (Chenofen, lives,?) in the knowledge graph, as shown in Figure 5 The triple is first taken as a to-be-predicted triple, and the text sequence corresponding to the to-be-predicted triple is: [CLS][Chenofen][SEP][lives][SEP][MASK][SEP]

[0117] [CLS][Chenofen][SEP][lives][SEP][MASK][SEP]

[0118] The above text sequence is input into the first entity representation model to obtain the representation vector corresponding to the Token [MASK]. The representation vector is used to query the knowledge repository to find K entities with the highest similarity to the representation vector, i.e., the K nearest neighbors of the representation vector, as candidate entities, which are assumed to be Greece, the United States, France, Germany, etc.

[0119] The representation vector corresponding to [MASK] and the information of the K candidate entities are input into the first representation mapping model to obtain the confidence of [MASK] in each candidate entity.

[0120] For each candidate entity, the similarity between the representation vector of the candidate entity and the representation vector corresponding to [MASK], and the confidence of [MASK] on the candidate entity are weighted to obtain a score of the candidate entity. In this way, the K candidate entities all have corresponding scores, and the candidate entity with the highest score "Greece" is selected as the prediction result.

[0121] The above entity prediction method provided by the embodiments of the present application can be applied to but is not limited to the following technical scenarios:

[0122] 1) Knowledge graph completion.

[0123] For a triple in which the head entity is known, the relationship between the head entity and the tail entity is known, but the tail entity is missing, the above method can be used to predict the missing tail entity as a to-be-predicted entity, and after obtaining the final prediction result, the entity corresponding to the prediction result is used to complete the missing tail entity in the knowledge graph.

[0124] For a triple in which the head entity is known, the relationship between the head entity and the tail entity is known, but the tail entity is missing, the above method can be used to predict the missing tail entity as a to-be-predicted entity, and after obtaining the final prediction result, the entity corresponding to the prediction result is used to complete the missing tail entity in the knowledge graph.

[0125] 2) Search recommendation scenario.

[0126] When a user inputs a query (search term) "In which country did Xenophon live" in a search engine, the above method provided by the embodiments of the present application can be used to predict the tail entity in (Xenophon, live,?) as a to-be-predicted triple. Then the tail entity is presented to the user in the search result page.

[0127] In this scenario, the knowledge graph can also be completed by using the method provided by the embodiments of the present application, and then the tail entity can be obtained by querying the knowledge graph when the user searches, and presented to the user in the search result page.

[0128] 3) Intelligent dialogue scenario.

[0129] In the intelligent dialogue scenario, when the user inputs a question "In which country did Xenophon live", the intelligent dialogue system can use the above method provided by the embodiments of the present application to predict the tail entity in (Xenophon, live,?) as a to-be-predicted triple. Then the tail entity is used to generate a reply and returned to the user.

[0130] In this scenario, the knowledge graph can also be completed by using the method provided by the embodiments of the present application, and then the tail entity can be obtained by querying the knowledge graph when the user uses the intelligent dialogue system to input a question, and the tail entity is used to generate a reply and returned to the user.

[0131] It can also be applied to other scenarios, which will not be listed here.

[0132] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0133] According to another embodiment, an entity prediction device is provided. Figure 6 A schematic block diagram of an entity prediction apparatus according to one embodiment is shown. Figure 6 As shown, the device 600 may include: a representation acquisition unit 601, a candidate query unit 602, a representation mapping unit 603, and a result determination unit 604, and may further include: a model training unit 605 and a repository construction unit 606. The main functions of each component are as follows:

[0134] The representation acquisition unit 601 is configured to input the text sequence corresponding to the triple to be predicted into the first entity representation model, wherein the triple to be predicted includes a known entity, the relationship between the known entity and the entity to be predicted, and the entity to be predicted, to obtain the representation vector of the entity to be predicted.

[0135] The candidate query unit 602 is configured to use the representation vector of the entity to be predicted to query a pre-built knowledge warehouse to obtain entities whose similarity between the representation vector and the representation vector of the entity to be predicted meets a preset similarity requirement as candidate entities, wherein the knowledge warehouse contains the representation vectors corresponding to each entity.

[0136] The representation mapping unit 603 is configured to input the representation vector of the entity to be predicted and the candidate entities into the first representation mapping model to obtain the prediction confidence of the entity to be predicted on each candidate entity.

[0137] The result determination unit 604 is configured to determine the prediction result of the entity to be predicted from the candidate entities based on the prediction confidence of the entity to be predicted on each candidate entity.

[0138] As one of the implementable manners, the model training unit 605 is configured to obtain first training data, the first training data comprising a plurality of triple samples, the triple samples comprising two entities and a relationship between the two entities; train the first entity representation model and the first representation mapping model by using the first training data; wherein the first entity representation model is input with a first text sequence corresponding to the triple sample, one entity in the first text sequence corresponding to the triple sample is masked, the first entity representation model outputs a representation vector corresponding to the masked entity to the first representation mapping model, and the first representation mapping model outputs a prediction result of the masked entity by using the representation vector corresponding to the masked entity; a training target is to minimize a difference between the prediction result output by the first representation mapping model and the masked entity in the triple sample, and the first entity representation model adopts a pre-trained language model.

[0139] As one of the implementable manners, the warehouse construction unit 606 is configured to pre-construct the above knowledge warehouse in the following manner:

[0140] For each entity in the entity library, the following is performed: obtaining a triple containing the entity, masking the entity in a first text sequence corresponding to the triple, and inputting the first text sequence corresponding to the triple into the first entity representation model to obtain a representation vector of the entity and store it into the knowledge warehouse.

[0141] As another implementable manner, the warehouse construction unit 606 is configured to pre-construct the above knowledge warehouse in the following manner:

[0142] For each entity in the entity library or each entity extracted from the page, the following is performed: learning a representation vector of the entity from a description text containing the entity and the entity by using a pre-trained language model based on a prompt learning mechanism, and storing the representation vector of the entity into the knowledge warehouse.

[0143] The warehouse construction unit 606 can be specifically configured as:

[0144] obtaining second training data, the second training data comprising an entity and a description text of the entity;

[0145] training a second entity representation model and a second representation mapping model using the second training data, wherein an entity in the second training data is masked and a description text of the entity is filled into a preset prompt template to obtain a second text sequence, the second text sequence is input into the second entity representation model, the second entity representation model outputs a representation vector corresponding to the masked entity to the second representation mapping model, and the second representation mapping model outputs a prediction result of the masked entity using the representation vector corresponding to the masked entity; a training target is to minimize a difference between the prediction result output by the second representation mapping model and the masked entity; and the second entity representation model uses a pre-trained language model;

[0146] After the training, a representation vector of each entity in the second training data is obtained from the second entity representation model and stored in the knowledge base.

[0147] It should be noted that the above two construction methods of the knowledge base can be used alone or in combination, for example, the latter method is used as a supplement to the former method.

[0148] The entity and the description text of the entity can be extracted from a Wikipedia page, a product detail page, or an official website page.

[0149] As one of the implementable ways, the prompt template includes a slot of the masked entity, a slot of the description text of the entity, and prompt information; and the prompt information is used to semantically prompt the content type to be predicted by the masked part.

[0150] As one of the implementable ways, the result determination unit 604 can be specifically configured to: for each candidate entity, respectively weight the similarity between the representation vector of the candidate entity and the representation vector of the entity to be predicted and the prediction confidence of the entity to be predicted on the candidate entity to obtain a score of the candidate entity; and select a candidate entity with the highest score from the candidate entities as the prediction result of the entity to be predicted.

[0151] It should be noted that the embodiments of the present application can involve the use of user data. In actual application, user-specific personal data can be used in the schemes described herein within the scope allowed by applicable laws and regulations, for example, with the explicit consent of the user, the user is actually notified, etc.

[0152] In addition, the embodiments of the present application also provide a computer readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps of the method of any one of the preceding method embodiments.

[0153] and an electronic device comprising:

[0154] one or more processors; and

[0155] a memory associated with the one or more processors, the memory for storing program instructions that, when read and executed by the one or more processors, perform the steps of the method of any one of the preceding method embodiments.

[0156] The present application also provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the method of any one of the preceding method embodiments.

[0157] wherein, Figure 7 An exemplary architecture of an electronic device is shown, which can specifically include a processor 710, a video display adapter 711, a disk drive 712, an input / output interface 713, a network interface 714, and a memory 720. The processor 710, the video display adapter 711, the disk drive 712, the input / output interface 713, the network interface 714, and the memory 720 can be communicatively connected through a communication bus 730.

[0158] The processor 710 can be implemented in the form of a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., for executing relevant programs to implement the technical solutions provided by the present application.

[0159] The memory 720 can be implemented in the form of a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, etc. The memory 720 can store an operating system 721 for controlling the operation of the electronic device 700, a basic input / output system (BIOS) 722 for controlling the low-level operation of the electronic device 700. In addition, a web browser 723, a data storage management system 724, and an entity prediction apparatus 725, etc. can also be stored. The entity prediction apparatus 725 can be an application program that specifically implements the above-mentioned steps in the embodiments of the present application. In general, when the technical solutions provided by the present application are implemented by software or firmware, the relevant program codes are stored in the memory 720 and executed by the processor 710.

[0160] The input / output interface 713 is configured to connect an input / output module to realize information input and output. The input / output module can be configured in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. The input device can include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device can include a display, a speaker, a vibrator, an indicator light, etc.

[0161] The network interface 714 is configured to connect a communication module (not shown in the figure) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired manner (such as a USB, a network cable, etc.) or a wireless manner (such as a mobile network, WIFI, Bluetooth, etc.).

[0162] The bus 730 includes a channel for transmitting information between various components (such as the processor 710, the video display adapter 711, the disk drive 712, the input / output interface 713, the network interface 714, and the memory 720) of the device.

[0163] It should be noted that although the above device only shows the processor 710, the video display adapter 711, the disk drive 712, the input / output interface 713, the network interface 714, the memory 720, and the bus 730, in the specific implementation process, the device can also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device can also only include the components necessary to implement the scheme of the present application, and does not have to include all the components shown in the figure.

[0164] From the above description of the embodiments, those skilled in the art can clearly understand that the present application can be realized by means of software and a necessary general hardware platform. Based on such understanding, the technical solutions of the present application can be embodied in the form of a computer program product, which can be stored in a storage medium such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments.

[0165] The various embodiments described in this specification are intended to be exemplary only. The same features and aspects of the various embodiments can be combined with each other, unless expressly stated otherwise. In particular, the description of the system or system embodiments is relatively brief, as the skilled person will readily understand the basic similarities between the system embodiments and the method embodiments. The system and system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e. may be located in one place or distributed over multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments. Those skilled in the art can understand and implement without creative labor.

[0166] The above describes the technical solutions provided by the present application in detail, and the principles and implementation modes of the present application are described by applying specific examples. The above description of the embodiments is only to help understand the method and its core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. An entity prediction method, characterized by, The method comprises: inputting a text sequence corresponding to a to-be-predicted triple into a first entity representation model, wherein the to-be-predicted triple comprises a known entity, a relationship between the known entity and a to-be-predicted entity, and the to-be-predicted entity, and a representation vector of the to-be-predicted entity is obtained; querying the representation vector in a pre-constructed knowledge repository to obtain an entity whose similarity to the representation vector of the to-be-predicted entity meets a preset similarity requirement as a candidate entity, wherein the knowledge repository comprises representation vectors corresponding to entities; inputting the representation vector of the to-be-predicted entity and the candidate entity into a first representation mapping model to obtain a prediction confidence of the to-be-predicted entity on each candidate entity; determining a prediction result of the to-be-predicted entity from the candidate entities according to the prediction confidence of the to-be-predicted entity on each candidate entity.

2. The method of claim 1, wherein, The first entity representation model and the first representation mapping model are pre-trained in the following manner: obtain first training data, wherein the first training data comprises a plurality of triple samples, and each triple sample comprises two entities and a relationship between the two entities; train the first entity representation model and the first representation mapping model using the first training data; wherein a first text sequence corresponding to the triple sample is input into the first entity representation model, one entity in the first text sequence is masked, the first entity representation model outputs a representation vector corresponding to the masked entity to the first representation mapping model, and the first representation mapping model outputs a prediction result of the masked entity using the representation vector corresponding to the masked entity; the training target is to minimize the difference between the prediction result output by the first representation mapping model and the masked entity in the triple sample, and the first entity representation model uses a pre-trained language model.

3. The method of claim 1, wherein, The knowledge repository is pre-constructed in the following manner: for each entity in the entity library, the following steps are performed: obtaining a triple comprising the entity, masking the entity in a first text sequence corresponding to the triple, and inputting the first text sequence corresponding to the triple into the first entity representation model to obtain a representation vector of the entity and store it in the knowledge repository.

4. The method according to claim 1 or 3, characterized in that, The knowledge repository is pre-constructed in the following manner: for each entity in the entity library or each entity extracted from a page, the following steps are performed: learning a representation vector of the entity from a description text comprising the entity and the entity using a pre-trained language model based on a prompt learning mechanism, and storing the representation vector of the entity in the knowledge repository.

5. The method of claim 4, wherein, The method comprises: obtain second training data, wherein the second training data comprises an entity and a description text of the entity; training a second entity representation model and a second representation mapping model using the second training data, wherein an entity in the second training data is masked and a description text of the entity is filled into a preset prompt template to obtain a second text sequence, the second text sequence is input into the second entity representation model, the second entity representation model outputs a representation vector corresponding to the masked entity to the second representation mapping model, the second representation mapping model outputs a prediction result of the masked entity using the representation vector corresponding to the masked entity; a training target is to minimize a difference between the prediction result output by the second representation mapping model and the masked entity; wherein the second entity representation model uses a pre-trained language model; after the training is completed, obtaining the representation vector of each entity in the second training data from the second entity representation model and storing the representation vector into the knowledge base.

6. The method of claim 5, wherein, the prompt template includes a slot of the masked entity, a slot of the description text of the entity, and prompt information; wherein the prompt information is used to semantically prompt the content type that needs to be predicted by the masked part.

7. The method of claim 1, wherein, determining the prediction result of the to-be-predicted entity from the candidate entities according to the prediction confidence of the to-be-predicted entity on each candidate entity includes: for each candidate entity, weighting the similarity between the representation vector of the candidate entity and the representation vector of the to-be-predicted entity and the prediction confidence of the to-be-predicted entity on the candidate entity to obtain a score of the candidate entity; selecting a candidate entity with the highest score from the candidate entities as the prediction result of the to-be-predicted entity.

8. An entity prediction apparatus characterized by comprising: The apparatus includes: a representation obtaining unit configured to input a text sequence corresponding to a to-be-predicted triple into a first entity representation model, wherein the to-be-predicted triple includes a known entity, a relationship between the known entity and a to-be-predicted entity, and the to-be-predicted entity, to obtain a representation vector of the to-be-predicted entity; a candidate query unit configured to query in a pre-constructed knowledge base using the representation vector of the to-be-predicted entity to obtain an entity whose similarity between the representation vector and the representation vector of the to-be-predicted entity meets a preset similarity requirement as a candidate entity, wherein the knowledge base includes a representation vector corresponding to each entity; a representation mapping unit configured to input the representation vector of the to-be-predicted entity and the candidate entity into a first representation mapping model to obtain a prediction confidence of the to-be-predicted entity on each candidate entity; a result determining unit configured to determine a prediction result of the to-be-predicted entity from the candidate entities according to the prediction confidence of the to-be-predicted entity on each candidate entity.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the method of any one of claims 1 to 7.

10. An electronic device, comprising: includes: one or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions, the program instructions being read and executed by the one or more processors to perform the steps of the method of any one of claims 1 to 7.

Images