Method for removing duplicate entities in a knowledge graph

Abstract

The present application relates to a method for deduplicating duplicate entities in a knowledge graph. By utilizing the entity information of entities in a training knowledge graph to construct multiple positive and / or negative examples as training data for adjusting a pre-training model, annotation costs are saved, and the entity deduplication goal is converted into a text binary classification task, which greatly reduces the difficulty of entity deduplication. There is no need to introduce quantitative calculation indicators. Under the condition that the number of positive and negative examples is large enough, the adjusted pre-training model is used as an entity deduplication model, and its deduplication accuracy will not decrease. On the contrary, it will be significantly improved compared with traditional deduplication methods.

Description

Technical Field

[0001] The present application relates to the field of library and information technology, and in particular to a method for removing duplicate entities in a knowledge graph. Background Art

[0002] Knowledge graphs are an important form of structured data storage. They use entities as their fundamental units and link entities through relationships. They are highly effective at storing entity attributes and relationships, and have important applications in various data storage and information retrieval applications. With the continuous implementation of artificial intelligence projects in various fields of Chinese, an increasing number of Chinese knowledge graph projects are being built, and the data volume of individual knowledge graphs is also growing.

[0003] As the volume of knowledge graphs increases, managing the quality of knowledge graph data becomes a thorny issue. One of the important issues is how to deduplicate and fuse duplicate entities in the knowledge graph.

[0004] The traditional method for deduplicating duplicate entities is an unsupervised entity deduplication algorithm. It first uses the "name" to recall the entities with the same name to be deduplicated in the knowledge graph. Then, based on the corresponding attributes of the two entities, it uses rules (such as whether the genders are the same, whether there is a hierarchical inclusion relationship between the birthplaces, etc.), edit distance (a method of expressing similar texts) and vector representation (Word2Vec, ELmo, etc.) to calculate the similarity between the two entities. Then, it performs weighted averaging, logical filtering, or sets a threshold on the similarity to determine whether the two entities are the same entity.

[0005] However, the main drawback of this approach is that, because it relies on calculating quantitative metrics to determine whether two entities are identical, it requires configuring and maintaining numerous rules for different entity types before deduplication can be performed. For example, for "poetry" or "province, city, and county" in the context of a location, these require additional manual organization and rule addition. Furthermore, specialized entity types require consultation with domain experts for professional advice. The limited universality of deduplication methods undoubtedly increases the difficulty of promoting and implementing such methods within knowledge graphs. Summary of the Invention

[0006] Based on this, it is necessary to provide a method for deduplicating duplicate entities in knowledge graphs to address the problem that traditional methods for deduplicating duplicate entities in knowledge graphs require configuring and maintaining a large number of rules for different types of entities.

[0007] This application provides a method for removing duplicate entities in a knowledge graph, including:

[0008] Obtain the knowledge graph for training;

[0009] Extracting entity information of an entity in the training knowledge graph;

[0010] Constructing multiple positive examples and / or multiple negative examples based on entity information of the entity;

[0011] Returning to the step of extracting entity information of an entity in the training knowledge graph until multiple positive examples and / or multiple negative examples are constructed for each of a preset number of entities in the training knowledge graph;

[0012] Inputting all positive examples and negative examples into a pre-trained model as training data, and training the pre-trained model to adjust the pre-trained model into an entity deduplication model;

[0013] Obtain a knowledge graph to be deduplicated, and select multiple entities to be deduplicated with the same name from the knowledge graph to be deduplicated;

[0014] Import the multiple entities with the same name to be deduplicated into the entity deduplication model, and run the entity deduplication model to deduplicate the entities in the knowledge graph to be deduplicated.

[0015] The present application relates to a method for deduplicating duplicate entities in a knowledge graph. By utilizing the entity information of entities in a training knowledge graph to construct multiple positive examples and / or negative examples as training data for adjusting a pre-training model, annotation costs are saved, and the entity deduplication goal is converted into a text binary classification task, which greatly reduces the difficulty of entity deduplication. There is no need to introduce quantitative calculation indicators. Under the condition that the number of positive and negative examples is large enough, the adjusted pre-training model is used as an entity deduplication model, and its deduplication accuracy will not decrease. On the contrary, it will be significantly improved compared with traditional deduplication methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flowchart of a method for removing duplicate entities from a knowledge graph provided in one embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0018] This application provides a method for removing duplicate entities in a knowledge graph.

[0019] like Figure 1 As shown, in one embodiment of the present application, the method for removing duplicate entities in the knowledge graph includes the following steps S100 to S700:

[0020] S100, obtaining a knowledge graph for training.

[0021] Specifically, the role of the training knowledge graph is to train the pre-trained model.

[0022] S200: Extract entity information of an entity in the training knowledge graph.

[0023] Specifically, the entity information includes one or more of a name (name), a unique identifier (id), a time (time), tags (tags), a brief introduction (summary) and a content introduction (content).

[0024] S300: Construct multiple positive examples and / or multiple negative examples based on the entity information of the entity.

[0025] Specifically, an entity in the knowledge graph is displayed as a point on the knowledge graph, and the relationship between entities is represented by a line. In this step, a positive example is an entity with a high similarity to the entity, and a negative example is an entity with a low similarity to the entity.

[0026] The purpose of creating multiple positive examples based on the entity is to enable the pre-trained model to distinguish whether two entities with high similarity are the same entity after training. Similarly, the purpose of creating multiple negative examples based on the entity is to enable the pre-trained model to distinguish whether two entities with low similarity are the same entity after training.

[0027] S400, returning to S200, until multiple positive examples and / or multiple negative examples for each of the preset number of entities in the training knowledge graph are constructed.

[0028] Specifically, the number of entities that the pre-training model can train in a single training run is controllable and is a preset number. For example, if 10 entities need to be trained in this training run, and 2 positive examples and 2 negative examples are constructed for each entity, then S200 to S300 need to be executed 10 times.

[0029] S500: All positive examples and negative examples are input into a pre-training model as training data, and the pre-training model is trained to adjust the pre-training model to an entity deduplication model.

[0030] Specifically, the pre-trained language model can adopt the BERT model and the RoBERTa model. During the training process, a deep neural network with large parameters and a large amount of unsupervised text corpus (i.e., multiple positive examples and / or multiple negative examples) are used for model training, which can provide a certain understanding of the semantics expressed by the text content displayed in the corpus without the need for additional model training. This understanding is expressed as a vector distribution in a high-dimensional space, which can be used as an understanding of the text task by operating on the vector distribution for actual text tasks. Because pre-trained models usually use full-domain text data and training methods that are not targeted at downstream tasks, the pre-trained model and training method used in this step make the pre-trained model highly adaptable to downstream tasks.

[0031] S600: Obtain a knowledge graph to be deduplicated, and select multiple entities to be deduplicated with the same name from the knowledge graph to be deduplicated.

[0032] Specifically, the knowledge graph to be deduplicated is different from the training graph. The training knowledge graph is used to train the pre-trained model, while the knowledge graph to be deduplicated is the knowledge graph that needs to be deduplicated in actual applications. Of course, the knowledge graph to be deduplicated can be a subgraph of the training graph.

[0033] Therefore, when acquiring the S100 training graph, we should try to ensure that the number of entities in the training graph is large and the entity information is comprehensive, so as to ensure that the results of entity deduplication in the subsequent knowledge graph to be deduplicated are more accurate.

[0034] This step converts the graph alignment task into a text binary classification task. The entity deduplication model is trained by a pre-trained model and has the ability to deduplicate entities. It does not require the calculation of quantitative indicators. It only needs to perform a text binary classification task of "determining whether it is the same entity". This greatly improves the efficiency of the deduplication step and reduces the complexity of the deduplication step.

[0035] S700: Import the multiple entities to be deduplicated with the same name into the entity deduplication model, and run the entity deduplication model to dedupe the knowledge graph to be deduplicated.

[0036] In this embodiment, by utilizing the entity information data of entities in a training knowledge graph to construct multiple positive and / or negative examples as training data for adjusting the pre-training model, the labeling cost is saved, and the entity deduplication goal is converted into a text binary classification task, which greatly reduces the difficulty of entity deduplication. There is no need to introduce quantitative calculation indicators, and under the condition that the number of positive and negative examples is large enough, the adjusted pre-training model is used as an entity deduplication model, and its deduplication accuracy will not decrease. On the contrary, it will be significantly improved compared with traditional deduplication methods.

[0037] In one embodiment of the present application, after S200, the method for removing duplicate entities in the knowledge graph further includes:

[0038] S210: Convert the structured data in the entity information into unstructured data.

[0039] Specifically, the knowledge graph can be understood as a database, and the entity information stored in the knowledge graph contains a large amount of highly structured text information.

[0040] Entity information generally consists of structured data and unstructured data. For example, the entity information corresponding to attributes such as name, dynasty, and tag is structured data, while the entity information corresponding to attributes such as simple explanation and content description is unstructured data.

[0041] However, the input requirement of the pre-training model is unstructured natural language text description, so it is necessary to convert the structured data in the entity information of the knowledge graph into unstructured data.

[0042] Since the entity information of the same entity can be transformed into different natural language descriptions through different transformation methods, these different natural language descriptions are all natural language texts of the entity. Therefore, this application uses this feature to construct multiple positive examples and / or negative examples for model training, namely S300 in this application. In one embodiment of this application, S300 includes the following S311 to S312:

[0043] S311 , splitting the entity information of the entity into multiple partial information, each partial information corresponding to an attribute of the entity.

[0044] S312, selecting multiple different partial information to form different positive examples.

[0045] Specifically, the implementation of this embodiment is suitable for entities with rich and diverse entity information. In this way, the amount of partial information obtained by splitting is large, and when different amounts of partial information are selected to form positive examples, the overall characteristics of the entity can still be well represented.

[0046] For example, after the entity information of entity A is split, it generates "Li Moumou", "Gender: Male", "Date of Birth: August 1975", "Introduction: Deputy Secretary of the Standing Committee of XX Province, formerly served as...", "Resume: Graduated from Tsinghua University in 2000,..., served as Deputy Secretary of the Standing Committee of XX Province in 2020", a total of 5 parts of information.

[0047] Then we can generate 3 positive examples like this:

[0048] Positive example A1: Li XX, male, born in August 1975.

[0049] Positive example A2: Li XX, born in August 1975, is the deputy secretary of the Standing Committee of XX Province and formerly served as...

[0050] Positive example A3: Li Moumou, male, born in August 1975, graduated from Tsinghua University in 2000,..., served as deputy secretary of the Standing Committee of XX Province in 2020.

[0051] It can be seen that these three positive examples are different from each other, and they select multiple different partial information to form different positive examples, but they can all represent the overall characteristics of the entity very well.

[0052] The advantage of this embodiment is that a large number of positive examples are generated with a high degree of diversity.

[0053] In one embodiment of the present application, S300 includes S321 to S322:

[0054] S321 : Split the entity information of the entity into multiple partial information, each partial information corresponding to an attribute of the entity.

[0055] S322: Select a preset number of partial information, and arrange the preset number of partial information in different orders to form different positive examples.

[0056] Specifically, for example, after the entity information of entity B is split, "Ms. Li", "Gender: Male", and "Date of Birth: August 1975" are generated, which is a total of three parts of information.

[0057] Then we can generate two positive examples like this:

[0058] Positive example B1: Li Moumou, male, born in August 1975.

[0059] Positive example B2: Mr. Li, born in August 1975, male.

[0060] It can be seen that the preset number is 2, and the different orders of the two partial information of the two positive examples produce two different positive examples.

[0061] In one embodiment of the present application, S300 includes the following S331 to S332:

[0062] S331 : Split the entity information of the entity into multiple partial information, each partial information corresponding to an attribute of the entity.

[0063] S332, retrieve multiple entity templates from the server, align all partial information with different entity templates, and output different positive examples.

[0064] Specifically, the server is provided with an entity template library, which stores multiple entity templates. This embodiment can extract the entity template and partial information for alignment, thereby generating a positive example. This implementation method is relatively time-saving and highly efficient.

[0065] For example, after the entity information of entity C is split, it generates "Ms. Li", "Gender: Male", and "Date of Birth: August 1975", a total of three parts of information.

[0066] Recalling entity template I: [partial information]-[partial information]-[partial information] and entity template II: [attribute + partial information]-[attribute + partial information]-[attribute + partial information] from the entity template library in the server, we can generate two positive examples like this:

[0067] Positive example C1: Li Moumou, male, born in August 1975.

[0068] Positive example C2: [Name] Li Moumou, [Gender] Male, [Date of Birth] Born in August 1975.

[0069] It can be seen that these two positive examples are generated using different entity templates.

[0070] In one embodiment of the present application, the S300 includes the following S341 to S345:

[0071] S341: Split the entity information of the entity into multiple partial information, each partial information corresponding to an attribute of the entity.

[0072] S342: Retrieve at least one necessary attribute tag of the entity from the server.

[0073] S343: Searching for multiple replaceable entries corresponding to each necessary attribute tag from the server according to all necessary attribute tags of the entity.

[0074] S344: Based on one or more necessary attribute tags of the entity, replace part of the information corresponding to the necessary attribute tags with a replaceable term to generate a counterexample.

[0075] S345 returns to S344 to generate multiple counterexamples, each with different replaceable terms. Specifically, the server also has a necessary attribute tag library. During the training process of the pre-trained model, necessary attribute tags are added to each entity. After training, the necessary attribute tags of each entity are stored in the necessary attribute tag library, and each necessary attribute tag has multiple replaceable terms added.

[0076] For example, after the entity information of entity D is split, it generates "Ms. Li", "Gender: Male", and "Date of Birth: August 1975", a total of three parts of information.

[0077] After retrieving the required attribute tags for entity D from the server, we found that of the three attributes (name, gender, and date of birth), gender and date of birth are required for entity D. The required attribute of date of birth has three alternative entries: August 1975, June 1845, and August 1945. The required attribute of gender has two alternative entries.

[0078] Then we can generate 6 counterexamples like this:

[0079] Counterexample D1: Li, male, born in August 1975. Counterexample D2: Li, male, born in June 1845. Counterexample D3: Li, male, born in August 1945.

[0080] Counterexample D4: Li, male, born in August 1975. Counterexample D5: Li, male, born in June 1845. Counterexample D6: Li, male, born in August 1945.

[0081] In one embodiment of the present application, S300 includes the following S351 to S353:

[0082] S351: Take the entity as a first entity, and split the entity information of the first entity into multiple first parts of information, each of which corresponds to an attribute of the first entity.

[0083] S352: Select another entity in the knowledge graph to be deduplicated as a second entity, extract entity information of the second entity, and split the entity information of the second entity into multiple second parts of information, each of which corresponds to an attribute of the second entity.

[0084] S353, obtaining the first part of information corresponding to the name attribute and the second part of information corresponding to the name attribute, swapping the first part of information corresponding to the name attribute with the second part of information corresponding to the name attribute to generate two counterexamples.

[0085] Specifically, for example, after the entity information of the first entity E is split, it generates "Bai", "Label: Poet", and "Dynasty: Tang Dynasty", a total of three parts of information. After the entity information of the second entity F is split, it generates "Li Moumou", "Gender: Male", and "Date of Birth: August 1975", a total of three parts of information.

[0086] Then we can generate two counterexamples like this:

[0087] Counterexample E1: Li Bai, male, born in August 1975.

[0088] Counterexample E2: Li XX, of the Tang Dynasty, a poet.

[0089] It can be seen that for these two counterexamples, except for the name attribute part of the entity information of the two entities, the remaining entity information is swapped. Adding such corpora can enable the pre-trained model after training to distinguish between the two significantly different entities of "Li XX" and "Li Bai".

[0090] It can be understood that the two entities selected in this embodiment need to have relatively rich entity information so as to show the obvious differences in the entity information parts of the two entities.

[0091] In an embodiment of the present application, S300 includes the following S361 to S362:

[0092] S361, splitting the entity information of the entity into multiple partial information, and each partial information corresponds to an attribute of the entity.

[0093] S362, retrieving different algorithms for streamlining partial information, streamlining all partial information, and generating different positive examples.

[0094] Specifically, the algorithm for streamlining partial information can filter out useless information in the partial information, such as modal particles, auxiliary words, and so on.

[0095] For example, after splitting the entity information of entity F, "Li XX", "Gender: Male", and "Date of Birth: August 1975" are generated, a total of 3 partial information.

[0096] After streamlining all partial information using the first algorithm for streamlining partial information, positive example F1 is generated: Li XX, male, born in August 1975.

[0097] After streamlining all partial information using the second algorithm for streamlining partial information, positive example F2 is generated: Li XX, male, August 1975.

[0098] Training the pre-trained model with the positive examples in this embodiment can enable the pre-trained model after training to recognize the relatively streamlined partial information in the entity information, ensuring the accuracy and stability of entity deduplication.

[0099] In an embodiment of the present application, S700 includes the following S710 to S770:

[0100] S710, creating a set of entities with the same name.

[0101] S720, selecting a name.

[0102] S730: Search all entities with the name from the knowledge graph to be deduplicated as entities to be deduplicated, and include all entities to be deduplicated into the entity set with the same name.

[0103] S740: Select an entity to be deduplicated from the set of entities with the same name as the main comparison entity.

[0104] S750: Select another entity to be deduplicated from the set of entities with the same name as a secondary comparison entity, and import the primary comparison entity and the secondary comparison entity into an entity deduplication model.

[0105] S760: Run the entity deduplication model, and use the entity deduplication model to compare and determine whether the primary comparison entity and the secondary comparison entity are the same entity.

[0106] S770, if the main comparison entity and the secondary comparison entity are not the same entity, return to S750 until the main comparison entity and all other entities to be deduplicated in the entity set with the same name have been compared and determined, and return to S740 until every two entities to be deduplicated in the entity set with the same name have been compared and determined once to see if they are the same entity.

[0107] Specifically, for example, the name "Li Bai" is selected. There are a total of 5 entities named "Li Bai" in the knowledge graph to be deduplicated. Then these 5 entities are all entities to be deduplicated and are included in the entity set K with the same name, in the form of K = {K1, K2, K3, K4, K5}.

[0108] Next, select K1 as the primary comparison entity and K2 as the secondary comparison entity. These entities are then imported into the entity deduplication model. Since the entity deduplication model is trained, it can automatically determine whether K1 and K2 are the same entity. After this determination, K3 is added as the secondary comparison entity, and the entity deduplication model again determines whether K1 and K3 are the same entity. This process continues until K1 has been compared with each of K2, K3, K4, and K5.

[0109] Next, K2 is used as the primary comparison entity. Since K1 and K2 have already been compared, K3 is used as the secondary comparison entity. The entity deduplication model determines whether K2 and K3 are the same entity. K2 and K4 are then compared to determine whether they are the same entity. This process is repeated until every two entities to be deduplicated in K1, K2, K3, K4, and K5 have been compared once to determine whether they are the same entity. The process ends.

[0110] In one embodiment of the present application, after S760, S700 further includes the following S780 to S790:

[0111] S780: If the primary comparison entity and the secondary comparison entity are the same entity, perform entity fusion on the primary comparison entity and the secondary comparison entity to generate a fused entity.

[0112] S790: After the entities are merged, return to S750.

[0113] Specifically, after the main comparison entity and the secondary comparison entity are fused, a new fused entity is generated in the knowledge graph to be deduplicated, and the main comparison entity and the secondary comparison entity before the fusion are deleted from the knowledge graph to be deduplicated.

[0114] The previous paragraph only describes the most basic fusion strategy. The detailed fusion strategies of S760 to S790 are relatively complex and can be implemented in multiple ways. An optional implementation method: For all entities to be deduplicated in the entity set with the same name, arbitrarily select two entities to be deduplicated to determine whether they are the same entity. If they are the same entity, the entities are fused to generate a new fused entity, and the number of entities to be deduplicated in the entity set with the same name is reduced by one. If the two entities to be deduplicated are not the same entity, both entities to be deduplicated are retained. The judgment is repeated in this way until each entity to be deduplicated in the entity set is not the same entity as the other entities to be deduplicated or there is only one entity to be deduplicated left in the set.

[0115] For example, if the set of entities with the same name is {K1, K2, K3, K4, K5}, first determine whether entities K1 and K2 are the same entity. If so, they are merged into entity Z1, and K1 and K2 are deleted from the set of entities with the same name. Next, determine whether entities Z1 and K3 are the same entity. If so, merge Z1 and K3 into entity Z2, and delete K3 from the set of entities with the same name. Further determine whether entities Z2 and K3 are not the same entity. If so, retain both Z2 and K3. Further determine whether entities Z2 and K4 are the same entity. If so, retain both Z2 and K4. Finally, determine whether entities K3 and K4 are the same entity. If so, merge K3 and K4 into entity Z3.

[0116] The entities that are ultimately retained are entities Z2 and Z3, resulting in a set of entities with the same name, [Z2, Z3]. The knowledge graph to be deduplicated is then modified based on the set of entities with the same name, generating Z2 and Z3 in the knowledge graph to be deduplicated, and deleting entities K1, K2, K3, K4, and K5.

[0117] In one embodiment of the present application, the S780 includes the following S781 to S789:

[0118] S781: Acquire entity information of the primary comparison entity and the secondary comparison entity respectively.

[0119] S782: Split the entity information of the primary comparison entity into a plurality of third parts of information, and split the entity information of the secondary comparison entity into a plurality of fourth parts of information.

[0120] S783, read the attributes of each third part of information and the attributes of each fourth part of information.

[0121] S784: Extract attributes that both the primary comparison entity and the secondary comparison entity have as common attributes.

[0122] S785: Determine whether the third part of information and the fourth part of information under the same common attribute are the same.

[0123] S786a, if the third part of the information and the fourth part of the information under the same common attribute are the same, then one of the third part of the information and the fourth part of the information is retained and the other is deleted.

[0124] S786b, if the third part of the information and the fourth part of the information under the same common attribute are different, then the one with more total characters is selected and retained, and the other is deleted.

[0125] S787: Extract the attribute that the primary comparison entity has but the secondary comparison entity does not have, and retain the third part of information corresponding to the attribute.

[0126] S788: Extract the attribute that the secondary comparison entity has but the primary comparison entity does not have, and retain the fourth part of information corresponding to the attribute.

[0127] S789: Recombine all the retained third part information and / or fourth part information to generate a fused entity.

[0128] Specifically, the core idea of this embodiment is to seek common ground while reserving differences, and to keep entity information concise in the process of seeking common ground. When some information is the same, only one is retained. This ensures that the merged entity can maintain the original characteristics of the two entities before the fusion.

[0129] The various technical features of the above-described embodiments can be combined arbitrarily, and the execution order of the method steps is not restricted. In order to make the description concise, not all possible combinations of the various technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0130] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for removing duplicate entities in a knowledge graph, characterized in that: The method comprises: Obtain the knowledge graph for training; Extracting entity information of an entity in the training knowledge graph and converting structured data in the entity information into unstructured data; Based on the converted unstructured data, construct multiple positive examples and / or multiple negative examples in the following ways: A. Splitting the entity information of the entity into multiple partial information, each partial information corresponding to an attribute of the entity; b. Select multiple different partial information to form different positive examples; c. selecting a preset number of partial information and arranging the preset number of partial information in different orders to form different positive examples; d. Retrieve multiple entity templates from the server, align all partial information with different entity templates, and output different positive examples; e. Replace part of the information corresponding to the necessary attribute labels with replaceable terms to generate counterexamples; Returning to the step of extracting entity information of an entity in the training knowledge graph until multiple positive examples and / or multiple negative examples are constructed for each of a preset number of entities in the training knowledge graph; Inputting all positive examples and negative examples into a pre-trained model as training data, and training the pre-trained model to adjust the pre-trained model into an entity deduplication model; Obtain a knowledge graph to be deduplicated, and select multiple entities to be deduplicated with the same name from the knowledge graph to be deduplicated; Importing the multiple entities to be deduplicated with the same name into the entity deduplication model, and running the entity deduplication model to dedupe the knowledge graph to be deduplicated, including: Create a set of entities with the same name, and select entities with the same name to compare with other entities in pairs; If they are determined to be the same entity, entity fusion is performed. During the fusion, the information with the largest total number of characters in the shared attributes is retained, and the non-shared attributes are integrated.

2. The method for removing duplicate entities in a knowledge graph according to claim 1, characterized in that: The constructing of multiple positive examples and / or multiple negative examples includes: Splitting the entity information of the entity into multiple partial information, each partial information corresponding to an attribute of the entity; Retrieving at least one required attribute tag of the entity from the server; Searching for multiple replaceable entries corresponding to each necessary attribute tag from a server according to all necessary attribute tags of the entity; According to one or more necessary attribute tags of the entity, part of the information corresponding to the necessary attribute tags is replaced with a replaceable term to generate a counterexample; Returning the one or more necessary attribute tags based on the entity, replacing part of the information corresponding to the necessary attribute tags with replaceable terms, and generating a counterexample step, so as to generate multiple counterexamples, each of which has a different replaceable term.

3. The method for removing duplicate entities in a knowledge graph according to claim 1, characterized in that: The constructing of multiple positive examples and / or multiple negative examples includes: Taking the entity as a first entity, splitting entity information of the first entity into multiple first parts of information, each first part of information corresponding to an attribute of the first entity; Selecting another entity in the knowledge graph to be deduplicated as a second entity, extracting entity information of the second entity, and splitting the entity information of the second entity into multiple second parts of information, each second part of information corresponding to an attribute of the second entity; The first part of information corresponding to the name attribute is obtained, and the second part of information corresponding to the name attribute is obtained. The first part of information corresponding to the name attribute and the second part of information corresponding to the name attribute are swapped to generate two counterexamples.

4. The method for removing duplicate entities in a knowledge graph according to any one of claims 2 to 3, characterized in that: Importing the multiple entities to be deduplicated with the same name into the entity deduplication model, and running the entity deduplication model to dedupe the knowledge graph to be deduplicated, includes: Create an entity collection with the same name; Choose a name; Search all entities with the name from the knowledge graph to be deduplicated as entities to be deduplicated, and include all entities to be deduplicated into the entity set with the same name; Select an entity to be deduplicated from the set of entities with the same name as the main comparison entity; Selecting another entity to be deduplicated from the set of entities with the same name as a secondary comparison entity, and importing the primary comparison entity and the secondary comparison entity into an entity deduplication model; Running the entity deduplication model, and using the entity deduplication model to compare and determine whether the primary comparison entity and the secondary comparison entity are the same entity; If the primary comparison entity and the secondary comparison entity are not the same entity, then return to the step of selecting another entity to be deduplicated in the set of entities with the same name as the secondary comparison entity, until the primary comparison entity and all other entities to be deduplicated in the set of entities with the same name have been compared and determined, and then return to the step of selecting an entity to be deduplicated in the set of entities with the same name as the primary comparison entity, until every two entities to be deduplicated in the set of entities with the same name have been compared and determined once to see if they are the same entity.

5. The method for removing duplicate entities in a knowledge graph according to claim 4, characterized in that: After running the entity deduplication model and comparing and determining whether the primary comparison entity and the secondary comparison entity are the same entity using the entity deduplication model, the method further includes: If the primary comparison entity and the secondary comparison entity are the same entity, performing entity fusion on the primary comparison entity and the secondary comparison entity to generate a fused entity; After the entities are merged, the process returns to the step of selecting another entity to be deduplicated from the set of entities with the same name as the secondary comparison entity.

6. The method for removing duplicate entities in a knowledge graph according to claim 5, characterized in that: The entity fusion of the primary comparison entity and the secondary comparison entity to generate a fused entity includes: Respectively obtaining entity information of the primary comparison entity and the secondary comparison entity; Splitting the entity information of the primary comparison entity into a plurality of third parts of information, and splitting the entity information of the secondary comparison entity into a plurality of fourth parts of information; Read the attributes of each third part of information and each fourth part of information; Extracting attributes possessed by both the primary comparison entity and the secondary comparison entity as common attributes; Determine whether the third part of information and the fourth part of information under the same common attribute are the same; If the third and fourth parts of information under the same common attribute are the same, then either one of the third and fourth parts of information is retained and the other is deleted; If the third and fourth parts of information under the same common attribute are different, the one with more total characters is retained and the other is deleted; Extracting an attribute possessed by the primary comparison entity but not possessed by the secondary comparison entity, and retaining the third part of information corresponding to the attribute; Extracting an attribute possessed by the secondary comparison entity but not possessed by the primary comparison entity, and retaining the fourth part of information corresponding to the attribute; All the retained third part information and / or fourth part information are recombined to generate a fused entity.

Images