repetition determination in the figure

Abstract

This invention discloses methods for determining repetition in graphs. Embodiments of the invention provide a method for determining repetition in a graph. The graph includes nodes representing entities and edges representing relationships between entities. Embodiments of the invention identify two target nodes in the graph. A neighborhood subgraph can be determined for each of the two nodes. The neighborhood subgraph includes the corresponding node. Embodiments of the invention can then determine whether the two nodes are repetitions relative to each other based on a comparison between the two subgraphs and a comparison of the two target nodes within the subgraphs.

Description

Technical Field

[0001] This invention relates to the field of digital computer systems, and more specifically, to a method for performing duplicate determination in a drawing. Background Technology

[0002] Database systems can, for example, use graph structures for semantic queries, which have nodes, edges, and attributes to represent and store data. The graph associates data items in the repository with a set of nodes and edges, where edges represent relationships between nodes. These relationships allow data in the repository to be directly linked together and, in many cases, retrieved using a single operation. Summary of the Invention

[0003] Various embodiments provide methods, computer systems, and computer program products for determining repetition in figures as described in the independent claims. Advantageous embodiments are described in the dependent claims. Embodiments of the invention may be freely combined with each other if they are not mutually exclusive.

[0004] In one aspect, the present invention relates to a computer implemented for determining repetition in a graph, the graph comprising nodes representing entities and edges representing relationships between entities. The method includes: identifying (referred to as a first identification step) at least two target nodes in the graph; determining (referred to as a first determination step) a neighborhood subgraph of each of the two target nodes, the neighborhood subgraph including the corresponding target node; comparing (referred to as a first comparison step) the two neighborhood subgraphs; comparing (referred to as a second comparison step) the two target nodes; and determining (referred to as the second determination step) whether the two target nodes are repetitions relative to each other based on the comparison of the two neighborhood subgraphs and based on the comparison of the target nodes.

[0005] In another aspect, the present invention relates to a computer program product including a computer-readable storage medium having computer-readable program code embodied therein, the computer-readable program code being configured to implement all the steps of the method according to the foregoing embodiments.

[0006] In another aspect, the present invention relates to a computer system for determining repetition in a graph, the graph comprising nodes representing entities and edges representing relationships between entities. The computer system is configured to: identify at least two target nodes in the graph; determine a neighborhood subgraph for each of the two target nodes, the neighborhood subgraph including the corresponding target node; compare the two neighborhood subgraphs; compare the two target nodes; and determine, based on the comparison of the two neighborhood subgraphs and based on the comparison of the target nodes, whether the two target nodes are repetitions relative to each other. Attached Figure Description

[0007] The embodiments of the present invention will now be explained in more detail by way of example and with reference to the accompanying drawings, wherein:

[0008] Figure 1A This is a diagram of a computer system based on an example of this topic.

[0009] Figure 1B This is a simplified diagram based on the example diagram of this topic.

[0010] Figure 2 This is a flowchart illustrating a method for repeatedly determining a diagram, based on an example from this topic.

[0011] Figure 3 Two subgraphs are shown, defined based on examples from this topic.

[0012] Figure 4 This is a flowchart illustrating a method for determining the first score, based on an example from this topic.

[0013] Figure 5 This is a flowchart illustrating a method for determining the first score, based on an example from this topic.

[0014] Figure 6 This refers to a computerized system that is adapted to implement one or more methodological steps as described in this topic. Detailed Implementation

[0015] The description of various embodiments of the present invention is presented for illustrative purposes and is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles of the embodiments, their practical application, or improvements to existing technologies in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0016] As used in this article, terms such as “first,” “second,” etc., are used as markers for the nouns preceding them and do not imply any type of order (e.g., spatial, temporal, logical) unless explicitly defined as such.

[0017] A graph refers to a property graph, where data values ​​are stored as attributes on nodes and edges. Property graphs can be managed and processed by a graph database management system or other database systems that provide a wrapper layer. This wrapper layer transforms the property graph into, for example, a relational table for storage and transforms the relational table back into a property graph when read or queried. The graph can be, for example, a directed graph. The graph can be a collection of nodes (also called vertices) and edges. An edge connects any two nodes in the graph. An edge can be represented by an ordered pair of nodes (v1, v2), and can be traversed from node v1 (the starting node) towards node v2. Nodes in a graph can represent entities. Entities refer to users, objects, etc. Entities can have certain entity properties or attributes that can be assigned values. For example, a person can be an entity. Entity attributes for a person can include, for example, marital status, age, gender, etc. Edge attribute values ​​can be assigned to edges, which at least indicate the relationship between the two nodes connected to that edge. This relationship can, for example, include inheritance (e.g., parent and child) relationships and / or association relationships based on a specific hierarchy. For example, the inheritance relationship between nodes v1 and v2 can be called an "is-a relationship" between v1 and v2, such as "v2 is the parent of v1". The association relationship between nodes v1 and v2 can be called a "has-a relationship" between v1 and v2, such as "v2 has a has-a relationship with v1" means that v1 is a part of v2, a component of v2, or associated with v2.

[0018] This graph can represent entities and relationships of a set of one or more domains. A domain (also called a domain ontology) represents a concept or category that belongs to a part of the world, such as biology or politics. Domains are typically modeled for domain-specific definitions of terms. For example, a domain can refer to specific fields such as healthcare, advertising, business, medicine, and / or biomedicine. A set of one or more domains can represent an ontology. For example, a set of domains could include a family domain and a company domain, where the family domain and the company domain belong to the same ontology, which could be an employee ontology.

[0019] However, processing graphs can be technically challenging because they typically have millions of nodes and edges, such as graphs storing millions of customer records, contracts, and personal records associated with companies with hundreds of thousands of employees. This can be particularly challenging in the case of data deduplication. For example, Master Data Management (MDM) systems use graphs as persistent storage to identify duplicate records and may need to parse them, if applicable. This process is a matching process and uses deterministic and / or probabilistic matching techniques with fuzzy operators, such as phonetics (Soundex, NYSIIS, ...), edit distance, nickname parsing, etc. This topic can improve the performance of such processing. To this end, the matching based on this topic can combine different scores obtained from independent comparisons of nodes in the surrounding subgraph. This can make it possible to identify entities as duplicates if the relational network between entities and their neighbors shows a high degree of similarity. In graph terminology, two records represented as nodes in the graph can be considered duplicates if they have high local subgraph similarity and high similarity between them. As a result, very important insights into whether two records should be merged are taken into account in the decision-making process that yields the best results. This approach can therefore improve the accuracy of the deduplication process. The following examples enable further improvements in the accuracy of the deduplication process.

[0020] According to one embodiment, a first comparison step of two neighborhood subgraphs generates a first score indicating the difference between the two neighborhood subgraphs. A second comparison of target nodes generates a second score indicating the similarity between target nodes. The method further includes: identifying (referred to as a second identification step) at least one pair of candidate nodes, such that one node in each pair is in one neighborhood subgraph and the other node in the pair is in another neighborhood subgraph, the candidate nodes being different from the target nodes; comparing (referred to as a third comparison step) the at least one pair of candidate nodes to generate at least one third score respectively indicating the similarity between the at least one pair of candidate nodes; reducing the first score based on the at least one third score; and combining the reduced first score and the second score to determine whether the two target nodes are duplicates relative to each other based on the combined score.

[0021] According to one embodiment, the first comparison step includes: performing a mapping between two neighborhood subgraphs, identifying a first pair of corresponding nodes that are not fully mapped to each other, identifying a second pair of corresponding nodes that are fully mapped to each other, assigning a second initial node score to each of the second pairs indicating that there is no difference between the pairs of nodes, assigning a first initial node score to each of the first pairs indicating that the pairs of nodes are completely different, and combining the first and second initial node scores to determine a first score indicating the comparison result of the two subgraphs.

[0022] According to one embodiment, the first comparison step includes: performing a mapping between two neighborhood subgraphs; identifying a first pair of corresponding nodes that are not fully mapped to each other; wherein at least one pair of identified candidate nodes is selected from the first pair; identifying a second pair of corresponding nodes that are fully mapped to each other; assigning a second initial node score to each pair in the second pair indicating that there is no difference between the pairs of nodes; assigning a first initial node score to each pair in the first pair indicating that the pairs of nodes are completely different; wherein reducing the first score includes: changing the first initial node score assigned to each pair of the at least one pair of identified candidate nodes based on a corresponding third score to indicate a reduced level of difference between the pairs of nodes; and combining the second initial node score, the unchanged first initial node score, and the changed first initial node score. The unchanged first initial node score may be the same as the original first initial node score.

[0023] According to one embodiment, a probabilistic matching method is used to perform the second and third comparison steps.

[0024] According to one embodiment, an entity has a set of entity attributes, wherein each of the second comparison step and the third comparison step includes: comparing the values ​​of the entity attribute sets of two compared nodes, and determining an attribute score for each attribute in the attribute set that indicates the comparison result of the attribute value, and combining the attribute scores to determine a first score or a third score.

[0025] According to one embodiment, the first identification step further includes determining the probability that two target nodes represent the same entity, and in response to determining that the probability is less than a threshold, performing a first determination, a comparison step, a second identification, and a second determination.

[0026] According to one embodiment, determining the neighborhood subgraph of the node includes selecting nodes of the graph using selection criteria. The selection criteria are based on at least one of the following: the number of nodes, the entities represented by the nodes, wherein the subgraph includes the selected nodes. This enables automatic and therefore efficient repeatable determination according to this topic.

[0027] According to one embodiment, the selection criteria require at least one of the following: the number of nodes in the subgraph is less than a maximum number; and the edges of the subgraph are connected to at least one node representing the same entity as the node. For example, the computer system may have configurable parameters associated with these criteria. These configurable parameters can be dynamically adjusted based on usage.

[0028] According to one embodiment, determining the neighborhood subgraph includes removing duplicate nodes from each subgraph of the neighborhood subgraph.

[0029] According to one embodiment, the method further includes using indications from the two identified nodes received for identification. For example, a user may provide input or a request. The received request or input includes information about the at least two nodes. This information can be used to identify at least two nodes in the graph.

[0030] Figure 1A An exemplary computer system 100 is depicted. The computer system 100 may be configured, for example, to perform master data management and / or data warehousing; for example, the computer system 100 may implement a deduplication system. The computer system 100 includes a data integration system 101 and one or more client systems 105 or data sources. Client systems 105 may include computer systems (e.g., as referenced). Figure 6 (As described). The data integration system 101 can control access to the graph database system 103 (read and write access, etc.).

[0031] The client system 105 can communicate with the data integration system 101 via a network connection, including, for example, a wireless local area network (WLAN) connection, a WAN (wide area network) connection, a LAN (local area network) connection, or a combination thereof.

[0032] Client system 105 can be configured to receive or generate query requests. For example, client system 105 generates or receives query requests at graph database system 103. Query requests may, for example, request the identification of repeating nodes. Client system 105 sends or forwards the query request to data integration system 101. Data integration system 101 can be configured to use graph database system 103 to extract data to compute an appropriate subset of graph 107 in graph database system 103, which is then sent back to client system 105 in response to the query request.

[0033] In another example, each client system 105 can be configured to send data records to data integration system 101 for storage by graph database system 103. A data record or record can be a collection of related data items, such as the name, date of birth, and category of a particular entity. A record represents an entity, where an entity refers to a user, object, or concept, and information about it is stored in the record. The terms "data record" and "record" are used interchangeably. Graph database system 103 can use graph 107 to store records as entities with relationships, where each record can be assigned to a node or vertex in graph 107, where attributes are attribute values ​​such as name, date of birth, etc. Data integration system 101 can use graph database system 103 to store records received from client system 105 and check for duplicate nodes in graph 107. For example, client system 105 can be configured to provide or create data records that may or may not have the same structure as graph 107. For example, client system 105 can be configured to provide records in XML or JSON format or other formats that can associate attributes with corresponding attribute values.

[0034] In one example, data integration system 101 may import data records from client system 105 using one or more extract-transform-load (ETL) batch processes or via Hypertext Transfer Protocol (“HTTP”) communication or other types of data exchange. Data integration system 101 and / or client system 105 may be associated with, for example, a personal computer (PC), a server, and / or a mobile device.

[0035] Data integration system 101 can be configured to process received records using one or more algorithms, such as algorithm 120 which implements at least a portion of the present method. For example, data integration system 101 can use algorithm 120 to process the data records of FIG107 in order to find record pairs or duplicate records representing the same entity according to the present disclosure. Although shown as separate components, in another example, graph database system 103 may be part of data integration system 101.

[0036] Figure 1BThis is a simplified diagram based on the exemplary domain ontology of Figure 107. The example ontology relates to employees of a company. Figure 107 has multiple nodes 130 representing company employees. Each node 130 may include attribute values ​​for entity attributes of the corresponding record. For example, the node 130 representing a wife has attribute values ​​for entity attribute 135, such as age, ID, etc. Nodes 130 are linked together by multiple edges 131. Edges 131 are associated with edge attribute values ​​137 that indicate the relationships between the various nodes. For example, the edge 131 linking node "Person 1" and node "Wife 1" is associated with the edge attribute value "Spouse," which indicates that the person represented by node "Person 1" is the spouse of the person represented by node "Wife 1." By traversing Figure 107, the data integration system 101 can identify which corresponding nodes are related to other nodes.

[0037] The diagram in this figure is represented as follows. Nodes in the diagram are represented by circles / ellipses, while edges are represented by lines connecting two nodes. Entities represented by nodes, such as "Wife 1" and "Person 1", are written inside circles. To simplify the diagram and description, only a portion of the nodes and edges, along with a portion of their associated entity and edge attributes, are shown. However, each node and each edge of the graph can be associated with one or more entity attributes and one or more edge attributes, respectively.

[0038] Figure 2 This is a flowchart illustrating a method for repetitive determination in Figure 107, based on an example of this topic. For illustrative purposes, the method can be described previously... Figure 1A This method is implemented in the computer system 100 shown in -B, but is not limited to this implementation. It can, for example, be executed by the data integration system 101.

[0039] In step 201, at least two nodes may be identified or determined in Figure 107. The identified nodes may be target nodes, which may need to be determined whether they represent the same entity, i.e., whether they are duplicate nodes. In one example, two nodes may be identified in step 201. In another example, more than two nodes may be identified in step 201. For example, the node identification in step 201 may be performed automatically, for instance, by randomly selecting two nodes from Figure 107. This may be advantageous when deduplication of the entire graph is required. In another example, node identification may be performed using information or requests received from a user. The received information or requests may indicate nodes. Those indicated nodes may be identified in step 201. In yet another example, node identification in step 201 may be performed in response to the insertion or addition of a new node in Figure 107, where the inserted node is part of the identified nodes. This enables controlled data storage as shown in Figure 107.

[0040] For the sake of simplicity, the number of nodes identified in step 201 may be two target nodes referred to as tnode1 and tnode2, but is not limited to this.

[0041] For each of the two target nodes, tnode1 and tnode2, a neighborhood subgraph of Figure 107 can be determined in step 203. This can generate a neighborhood subgraph including target node tnode1 and another neighborhood subgraph including target tnode2. The neighborhood subgraph of tnode1 in Figure 107 can be a subgraph of Figure 107 derived from the neighboring nodes of tnode1, for example, a subgraph consisting of the nodes adjacent to tnode1 and all edges connecting the nodes adjacent to tnode1. The neighboring nodes of node tnode1 in Figure 107 are nodes connected to tnode1 by edges. Figure 1B An example can be provided in step 203. Figure 3 The two subgraphs are 307A and 307B.

[0042] In one example, the user could be prompted to determine the neighborhood subgraph by displaying Figure 107 and the two identified nodes on the screen; for example, a zoomed-in view of Figure 107 around the two identified nodes could be shown. The user can provide input indicating the two neighborhood subgraphs.

[0043] In one example, one or more selection criteria or conditions can be used to automatically determine neighborhood subgraphs. For example, the subgraph of tnode1 should satisfy a first condition, according to which each edge of the subgraph connects to at least one node representing the same entity type as node tnode1. This at least one node can be, for example, the starting node of an edge. That is, the first condition can require that each edge of the graph has a starting node representing the same entity type as node tnode1. This makes it possible to obtain subgraphs representing the same domain, and therefore enables efficient subgraph mapping. However, the size of the resulting subgraph can be large, for example, the number of nodes in the subgraph exceeds a threshold. In this case, the first condition can be supplemented with a second condition, according to which the maximum distance between tnode1 and any node in the subgraph can be N edges, where, for example, N>=2. The distance between two nodes in the graph can be the number of edges in the shortest path connecting them. For example, if node adj1, a neighboring node of node tnode1, represents the same entity type as tnode1 and N=2, then the neighboring node adj3 of adj1 and their connecting edge (i.e., between adj1 and adj3) can be added to the subgraph. In other words, if adj1 represents the same entity type as node 1, the path from node tnode1 via node adj1 can include at most tnode1 and adj1 and adj3. Alternatively, if node adj1 represents an entity type different from the entity type represented by tnode1, the path from node tnode1 can include at most tnode1 and adj1. For example, if tnode1 represents a person and the neighboring node adj1 represents a company, then the path from tnode1 can include at most adj1. In another example, if tnode1 represents a person and the neighboring node adj1 also represents a person, then if N=2, the path from tnode1 can include adj1 and one other node adjacent to adj1. The determination of the subgraph of tnode2 is similar to the determination of the subgraph of tnode1 described above. Therefore, selection criteria enable the controllability and efficiency of subgraph selection, thereby achieving optimal subgraph mapping. The determined subgraph can be further improved by using one or more predefined filtering rules. For example, a first filtering rule might require that edges of predefined types be filtered out from the determined subgraph. A second filtering rule might require that nodes representing predefined entities be filtered out or assigned low weights that should be considered during subgraph mapping. Filtering rules can optionally be applied to the determined subgraph. Another example of determining a subgraph could include determining how to construct a machine learning classifier for the subgraph. The classifier can be trained based on administrator feedback.

[0044] In step 205, the identified subgraphs can be compared with each other. This can be referred to as the first comparison step. For example, a first score S can be determined based on the comparison results between two subgraphs. GRP First score S GRP This could indicate the probability that two subgraphs are different. For example, suppose two subgraphs contain J corresponding node pairs, where the J corresponding node pairs exclude the identified target node pair tnode1 and tnode2. For example, a comparison of the two subgraphs can be performed between the J corresponding node pairs. Each of the J corresponding node pairs can be fully mapped to each other or not fully mapped to each other. For example, J1 node pairs can be fully mapped, while the remaining J2 node pairs can not be fully mapped. When they are fully mapped, each of the J1 corresponding node pairs can be assigned an initial node score of zero, and when they are not fully mapped, each of the J2 corresponding node pairs can be assigned an initial node score of 1. These initial node scores can be summed to obtain a first score S. GRP For example, S GRP =J1×0+J2×1=J2. Figure 4 and 5 Provided first score S GRP The example is determined.

[0045] In step 209, the two target nodes tnode1 and tnode2 can be compared with each other. This can be referred to as the second comparison step. The comparison result of the two target nodes tnode1 and tnode2 can confirm or further indicate whether the two target nodes tnode1 and tnode2 are duplicates. The second comparison can be, for example, by comparing the values ​​of the entity attribute sets of the two target nodes individually and for each attribute ATT in the attribute set. i (i = 1, ..., n, where n is the number of attributes) Determine the indicator attribute ATT i The attribute score of the comparison result of the values To execute. Attribute score. These scores can be combined to determine a second score S, indicating the similarity between two target nodes, based on a second comparison. TNODE A combination of attribute scores can be, for example, the sum of attribute scores. In another example, the combination of attribute scores could be achieved using the weights w assigned to the entity attributes. i The weighted sum of attribute scores For example, when comparing two people, the last name can have a higher weight than attributes such as address.

[0046] In step 211, it can be determined whether the two target nodes tnode1 and tnode2 are duplicates relative to each other based on the results of the comparisons performed in step 205 and step 209. For example, if the first score S GRP Less than (i.e., below) the first threshold, and the second score S TNODE If the score is above the second threshold, the target node can be a duplicate node. In another example, the second score can be combined with the reciprocal of the first score (e.g., S). TNODE +1 / S GRP This is done to obtain a combined score, and if the combined score is higher than a threshold, the target node can be a duplicate node.

[0047] If it is determined that two nodes tnode1 and tnode2 are duplicate nodes, the two nodes tnode1 and tnode2 can be merged, and the correct edge embedding can be re-established in Figure 107.

[0048] Figure 3 The subject matter of the present invention is shown as respectively representing Figure 1B The two nodes “Person 1” and “Person 2” in Figure 107 define two subgraphs, 307A and 307B. Subgraph 307A, defined by node “Person 1”, has solid edges and nodes representing “Wife 1”, “Child 1”, “Child 2”, “Company”, and “Person 3”, while subgraph 307B, defined by node “Person 2”, has dashed edges and nodes representing “Wife 2”, “Child 1”, “Child 2”, “Company”, and “Person 4”.

[0049] The following selection criteria can be used to determine the two subgraphs: out-degree person-to-person = 2, out-degree person-to-company = 1. "Out-degree person-to-person" refers to the maximum distance of the path starting from the node representing a person and including adjacent nodes representing people. "Out-degree person-to-company" refers to the maximum distance of the path starting from the node representing a person and including adjacent nodes representing companies. This results in nodes such as "person5" and "person6" not being considered because the out-degree of person-to-company is 1, meaning the distance between nodes "person2" and "person6" is two, which is higher than the maximum distance (which is one), since the two first nodes of the path starting from node "person2" represent different entities. A filtering rule can also be used to determine the two subgraphs, according to which only person and company nodes are included in the subgraph.

[0050] Figure 4 This is based on examples from this topic used to determine and use the first score S. GRPThe flowchart of the method. For illustrative purposes, the method can be seen in the previous... Figure 1A This method is implemented in the computer system 100 shown in -B, but is not limited to this implementation. It can, for example, be executed by the data integration system 101.

[0051] In step 401, at least one pair of candidate nodes can be identified. For example, in step 401, K pairs of candidate nodes can be identified, where K ≥ 1. This can be referred to as the second identification step. The K pairs of candidate nodes can be selected from... Figure 2 The nodes are selected from the J2 pairs described in the diagram, where K ≤ J2. In one example, all J2 pairs can be identified in step 401, i.e., K = J2. In another example, only pairs of J2 pairs that appear substantially similar can be identified in step 401. (Following...) Figure 3 For example, each pair of nodes (wife 1, wife 2) can be the pair identified in step 401 (i.e., K=1) because it has a higher probability of being a similar node pair than another pair of nodes (person 3, person 4). This is because nodes "person 1" and "person 2" share the same child nodes.

[0052] In each of the K pairs, one node of each pair is in a neighborhood subgraph 307A, and the other node of the pair is in another neighborhood subgraph 307B. Candidate nodes are different from target nodes. According to... Figure 3 For example, the mapping of two subgraphs 307A and 307B can produce the following sets of first node pairs (person 1, person 2), (person 3, person 4), and (wife 1 and wife 2), and second node pairs (child 1, child 1), (child 2, child 2), and (company, company). K pairs of candidate nodes can be selected from the first pair sets (person 3, person 4) and (wife 1, wife 2). Figure 2 The example indicates the number of fully mapped node pairs J1 = 3, and the number of incompletely mapped node pairs J2 = 2, excluding the target node (person 1, person 2).

[0053] In step 403, K pairs of candidate nodes can be compared. This can be referred to as the third comparison step. This third comparison step can generate K third scores, each indicating the similarity between the K pairs of candidate nodes. (where j = 1...K).

[0054] First score S GRP You can first, as Figure 2 As described above, for example, using formula S GRP =J1×0+J2×1=J2 to determine. Additionally, a third score can be used in step 405. This can lead to a lower first score. First score S GRPThe reduction can be achieved by first based on the corresponding third score. The value will be associated with the initial node score of K for candidate nodes, which will be reduced by 1 to execute.

[0055] For example, if the third score of a pair of nodes Indicating a match between the nodes of the pair (e.g., 80% match), the initial node score can be significantly reduced; for example, 1 can be reduced by 80%*1. If the third score of a pair of nodes... If no match is found between the nodes of the pair, the initial node score can be maintained at 1. If a clerical case is found, the initial score can be reduced to a certain level, for example, by a user-defined reduction value. A clerical case refers to a document record. A document record is a record for which a given matching process cannot determine whether they are duplicates of each other and should therefore be merged, or if one or more should be considered non-matches and therefore should remain separate. These document records may require user intervention to examine the values ​​of the data records more closely.

[0056] In step 407, the reduced first score obtained using this method can be... With the second score S TNODE The scores are combined to determine whether two target nodes (e.g., target nodes that can be identified as described in step 201) are duplicates relative to each other. For example, the second score can be combined with the inverse of a reduced first score (e.g., In order to obtain a combined score, and if the combined score is higher than a threshold, the target node can be a duplicate node.

[0057] Figure 5 This is a flowchart illustrating a method for determining the first score, based on an example from this topic. For illustrative purposes, the method can be described previously. Figure 1A This method is implemented in the computer system 100 shown in -B, but is not limited to this implementation. It can, for example, be executed by the data integration system 101.

[0058] In step 501, a mapping between two neighborhood subgraphs can be performed.

[0059] In step 503, the first pair of corresponding nodes that are not completely mapped to each other can be identified. Based on the examples of subgraphs 307A and 307B, the mapping of the two subgraphs can produce the following first node pairs (wife1, wife2), (person1, person2), and (person3, person4).

[0060] In step 505, a second pair of corresponding nodes that are completely mapped to each other can be identified. Based on examples of subgraphs 307A and 307B, the mapping of the two subgraphs can produce the following second node pairs: (sub1, sub1), (sub2, sub2), (company, company). A first score can be provided, for example, as 3 / 6 = 0.5, indicating that 3 out of 6 pairs are identified as not being the same node. In another example, steps 507 through 511 can be used to determine the first score.

[0061] In step 507, an initial first node score can be assigned to each of the first pairs, indicating that the pair of nodes is completely different. For example, the initial first node score could be one. The number of the first pairs other than the target pair is J2.

[0062] In step 509, an initial second node score can be assigned to each of the second pairs, indicating that there is no difference between the nodes in that pair. For example, the initial second node score could be zero.

[0063] In one example, the initial first node score and the initial second node score can be combined to determine a first score indicating the comparison result of the two subgraphs. In another example, Figure 4 The K third scores calculated in step 511 can be used to modify the corresponding initial first node scores to update / change the difference level between the pair of nodes based on the third scores (obtained using the mapping in step 501). For example, if the initial first node scores are as defined above, such as equal to 1 for completely different nodes, then in step 511, the third scores can be used to reduce the corresponding initial first node scores (e.g., as...). Figure 4 (As described in [the original text]). In fact, the K third scores indicate the level of similarity between the corresponding node pairs of the J2 first node pairs (e.g., values ​​between 0% and 100%). The initial first node score assigned to each of the K node pairs can be used to reduce the third scores, for example, as follows: In step 513, the second initial node score, the unreduced initial first node score, and the reduced initial first node score can be combined to determine the first score.

[0064] Figure 6 This refers to a general computerized system 600 suitable for implementing at least a portion of the method steps as described in this disclosure.

[0065] It should be understood that the methods described herein are at least partially non-interactive and are automated through computerized systems such as servers or embedded systems. However, in exemplary embodiments, the methods described herein can be implemented in (partially) interactive systems. These methods can also be implemented in software 612, 622 (including firmware of BIOS 622), hardware (processor) 605, or combinations thereof. In exemplary embodiments, the methods described herein are implemented as software as an executable program and executed by a dedicated or general-purpose digital computer such as a personal computer, workstation, minicomputer, or mainframe computer. Thus, the most general system 600 includes a general-purpose computer 601.

[0066] In an exemplary embodiment, in terms of hardware architecture, such as Figure 6 As shown, computer 601 includes processor 605, memory (main memory) 610 coupled to memory controller 615, and one or more input and / or output (I / O) devices 645 (or peripherals) communicatively coupled via local input / output controller 635. Input / output controller 635 may be, but is not limited to, one or more buses or other wired or wireless connections, as known in the art. Input / output controller 635 may have additional elements to enable communication; these additional elements, such as controllers, buffers (caches), drivers, repeaters, and receivers, are omitted for simplicity. Furthermore, the local interface may include address, control, and / or data connections to enable proper communication between the aforementioned components. As described herein, I / O devices 10, 645 may generally include any general-purpose encryption card or smart card known in the art.

[0067] Processor 605 is a hardware device for executing software, specifically software stored in memory 610. Processor 605 can be any custom or commercially available processor, central processing unit (CPU), auxiliary processor among several processors associated with computer 601, semiconductor-based microprocessor (in the form of a microchip or chipset), or any device typically used for executing software instructions.

[0068] The memory 610 may include any one or a combination of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, etc.)) and non-volatile memory elements (e.g., ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM)). Note that the memory 610 may have a distributed architecture in which the various components are far apart from each other but can be accessed by the processor 605.

[0069] The software in memory 610 may include one or more separate programs, each program including an ordered list of executable instructions for implementing logical functions, particularly those involved in embodiments of the invention. Figure 6 In the example, the software in memory 610 includes instructions 612, such as instructions for managing a database, like a database management system.

[0070] The software in memory 610 typically also includes a suitable operating system (OS) 611. OS 611 essentially controls the execution of other computer programs, such as possible software 612 for implementing the methods described herein.

[0071] The method described herein can be in the form of source program 612, executable program (e.g., object code), script, or any other entity including a set of instructions to be executed. When it is a source program, it needs to be translated by a compiler, assembler, interpreter, etc., to operate correctly with OS 611, wherein the compiler, assembler, interpreter, etc., may or may not be included in memory 610. Furthermore, the method can be written in an object-oriented programming language with data classes and method classes, or a procedural programming language with routines, subroutines, and / or functions.

[0072] In an exemplary embodiment, a conventional keyboard 650 and mouse 655 may be coupled to an input / output controller 635. Other output devices, such as I / O device 645, may include input devices, such as, but not limited to, printers, scanners, microphones, etc. Finally, I / O devices 10, 645 may also include devices that communicate with input and output, such as, but not limited to, network interface cards (NICs) or modulators / demodulators (for accessing other files, devices, systems, or networks), radio frequency (RF) or other transceivers, telephone interfaces, bridges, routers, etc. I / O devices 10, 645 may be any general-purpose encryption card or smart card known in the art. System 600 may also include a display controller 625 coupled to a display 630. In an exemplary embodiment, system 600 may also include a network interface for coupling to a network 665. Network 665 may be an IP-based network for communication between computer 601 and any external server, client, etc., via a broadband connection. Network 665 sends and receives data between computer 601 and external system 30, and may be involved in performing some or all of the steps of the methods discussed herein. In an exemplary embodiment, network 665 may be a managed IP network managed by a service provider. Network 665 may be implemented wirelessly, for example using wireless protocols and technologies such as WiFi, WiMax, etc. Network 665 may also be a packet-switched network, such as a local area network (LAN), wide area network (WAN), metropolitan area network (MAN), the Internet, or other similar network environments. Network 665 may be a fixed wireless network, wireless local area network (WLAN), wireless wide area network (WWAN), personal area network (PAN), virtual private network (VPN), intranet, or other suitable network system, and includes devices for receiving and transmitting signals.

[0073] If the computer 601 is a PC, workstation, intelligent device, etc., the software in the memory 610 may also include a Basic Input / Output System (BIOS) 622. The BIOS is a set of basic software routines that initialize and test the hardware at startup, boot the OS 611, and support data transfer between hardware devices. The BIOS is stored in ROM so that it can be executed when the computer 601 is activated.

[0074] When computer 601 is running, processor 605 is configured to execute software 612 stored in memory 610, transfer data to and from memory 610, and generally control the operation of computer 601 according to the software. The methods and OS 611 described herein, in whole or in part, but generally the latter, are read by processor 605, may be cached in processor 605, and then executed.

[0075] When implementing the system and methods described herein in software 612, such as Figure 6As shown, the method can be stored on any computer-readable medium, such as storage device 620, for use by or in conjunction with any computer-related system or method. Storage device 620 may include a disk storage device such as an HDD storage device.

[0076] This invention can be a system, method, and / or computer program product at any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform aspects of the invention.

[0077] Computer-readable storage media can be tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or recessed structures with instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0078] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the respective computing / processing device.

[0079] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages ​​such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, to perform aspects of this invention, electronic circuits, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions to personalize the electronic circuits by utilizing the status information of the computer-readable program instructions.

[0080] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0081] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other devices to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored includes an article of writing comprising instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0082] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus or other device, perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0083] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions indicated in the blocks may occur in a different order than indicated in the figures. For example, two blocks shown consecutively may actually be implemented as a single step, executed simultaneously, substantially simultaneously, with partial or complete time overlap, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

Claims

1. A computer-implemented method for duplicate node identification for a graph database system, comprising: performing duplicate determination on a graph structure stored in the graph database system, the graph structure comprising nodes representing entities and edges representing relationships between the entities, wherein performing duplicate determination on the graph structure comprises: identifying at least two target nodes in the graph structure; determining a neighborhood subgraph for each of the two target nodes, the neighborhood subgraph comprising the respective target node, identifying at least one pair of candidate nodes such that one node in each pair of candidate nodes is in one neighborhood subgraph and the other node in the pair is in the other neighborhood subgraph, the at least one pair of candidate nodes being different from the target nodes; computing a first score of difference between the two neighborhood subgraphs; computing a second score of node similarity between the two target nodes; computing a third score of similarity between the at least one pair of candidate nodes; modifying the first score based on the third score; determining whether the two target nodes are duplicates with respect to each other based on the modified first score and the second score; and in the event that the two target nodes are identified as duplicates, merging the two target nodes and reestablishing correct edge embeddings.

2. The computer-implemented method of claim 1, wherein, wherein a lower first score indicates similarity between the two neighborhood subgraphs, and wherein a higher second score indicates similarity with the two target nodes.

3. The computer-implemented method of claim 2, further comprising: reducing the first score based on at least a third score; and combining the reduced first score and the second score to determine whether the two target nodes are duplicates with respect to each other.

4. The computer-implemented method of claim 3, further comprising: performing a mapping between the two neighborhood subgraphs; identifying a first pair of corresponding nodes that do not map perfectly to each other, wherein the at least one pair of identified candidate nodes is selected from the first pair of corresponding nodes; identifying a second pair of corresponding nodes that map perfectly to each other; assigning a first initial node score to each of the at least one pair of identified candidate nodes associated with the first pair of corresponding nodes, the first initial node score indicating that the respective pair of nodes are perfectly different; and assigning a second initial node score to each of the at least one pair of identified candidate nodes associated with the second pair of corresponding nodes, the second initial node score indicating that there is no difference between the respective pair of nodes.

5. The computer-implemented method of claim 3, wherein reducing the first score based on the at least a third score comprises: based on the respective third score, changing the first initial node score assigned to each of the at least one pair of identified candidate nodes to indicate a reduced level of difference between the pair of nodes; and combining the second initial node score, the unchanged first initial node score, and the changed first initial node score. computing a first score of difference between the two neighborhood subgraphs comprises:

6. The computer-implemented method of claim 1, wherein, performing a mapping between the two neighborhood subgraphs; identifying a first pair of corresponding nodes that do not map perfectly to each other; identifying a second pair of corresponding nodes that map perfectly to each other; ​ assigning a first initial node score to each of the first pairs of corresponding nodes, the first initial node score indicating that the first pair of nodes are completely different; assigning a second initial node score to each of the second pairs of corresponding nodes, the second initial node score indicating that there is no difference between the second pair of nodes; and combining the first initial node score and the second initial node score to determine a first score indicating a result of the comparison of the two sub-graphs.

7. The computer-implemented method of claim 3, wherein, The third score is calculated using a probabilistic matching method.

8. The computer-implemented method of claim 3, wherein each pair of nodes in the pair of candidate nodes corresponds to an entity having a set of entity attributes, and wherein calculating the third score of the similarity between at least one pair of candidate nodes further comprises: comparing values of the set of entity attributes of the two compared nodes, determining, for each attribute in the set of entity attributes, an attribute score indicating a result of the comparison of the values associated with the respective attribute, and combining the attribute scores to determine the third score.

9. The computer-implemented method of claim 1, further comprising: determining a probability that the two target nodes are duplicates with respect to each other; and in response to determining that the probability is higher than a threshold value, determining, for each of the two target nodes, a neighborhood sub-graph, the neighborhood sub-graph including the respective target node. Determining, for each of the two target nodes, a neighborhood sub-graph comprises:

10. The computer-implemented method of claim 1, wherein, selecting nodes of the graph using a selection criterion, wherein the selection criterion is based on at least one of: a number of nodes, an entity represented by a node, a distance between the node and another node in the sub-graph; and the sub-graph includes the selected nodes. The selection criterion requires at least one of:

11. The computer-implemented method of claim 10, wherein, a number of nodes of the sub-graph is less than a maximum number; an edge of the sub-graph connects to at least one node representing an entity that is the same as an entity of the node; and a distance between the node and another node in the sub-graph is less than a threshold number of edges.

12. The computer-implemented method of claim 1, wherein determining a neighborhood sub-graph for each of the two target nodes further comprises: removing duplicate nodes of each of the neighborhood sub-graphs.

13. The computer-implemented method of claim 1, further comprising: identifying at least two target nodes in the graph using the received indication of the two identified nodes.

14. A computer program product comprising: one or more computer-readable storage media and program instructions stored on the one or more computer-readable storage media, the program instructions executable by a processing unit to cause the processing unit to perform the operations of the method of any one of claims 1 to 13.

15. A computer system comprising: one or more computer processors; one or more computer-readable storage media; and ​ program instructions stored on the one or more computer-readable storage media for execution by at least one of the one or more computer processors, which, as a result of being executed, enable the one or more computer processors to carry out operations of the method according to any one of claims 1 to 13.

Images