Real-time solving in identity graph data structures

Abstract

A system and method for incrementally resolving entities without having to load and analyze the entire identity graph uses a subset, or block, of the graph. A data stream containing individual events is received from a message bus. The events are read and parsed, and data entities are constructed from the data events. A set of prediction rules is applied to identify blocks of the identity graph that are likely to be modified by new data events. Data events are matched against data in blocks of the identity graph to perform data entity resolution. This incremental approach significantly reduces the number of comparisons made to find a match, making resolution fast enough to function in real time or near real time, even for very large identity graphs.

Description

[Technical Field]

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 412,962, filed October 4, 2022, which is incorporated herein by reference in its entirety. [Background technology]

[0002] An identity graph is a data structure that encompasses data about a large collection of objects, such as consumers. Nodes in the identity graph correspond to the contacts or data elements of such objects, and edges in the graph connect nodes that correspond to the same object with a certain probability. Entity resolution (ER) refers to techniques and tools for identifying and linking different representations of the same object. ER is alternatively referred to as record linking, duplicate detection, or object consolidation. ER techniques may be used to identify and link objects in an identity graph, such as nodes in the identity graph that are associated with the same consumer in a set of consumers.

[0003] As identity graph operators acquire data about new objects or about a particular existing set of objects, the identity graph in which this information is stored inevitably grows. The number of nodes and the number of edges connecting these nodes continue to increase. As the identity graph grows, the number of edges that need to be traversed and the number of nodes that need to be examined during ER operations increases, so the time required to perform entity resolution within the identity graph is generally a function of the size of the identity graph. Thus, collecting more data slows down entity resolution using traditional search and matching algorithms.

[0004] In certain applications, it is desirable to perform ER operations in real time or near real time. Therefore, as the size of the identity graph increases, problems arise because the identity graph may become too large to perform the ER operations within the required time frame. Therefore, to support certain real-world applications, particularly applications that require real-time or near-real-time ER operations, it is desirable to develop systems and methods for performing ER operations within an identity graph within a short time frame, even when the identity graph becomes very large in size.

[0005] The references mentioned in this background section are not admitted to be prior art with respect to the present invention. Summary of the Invention [Means for solving the problem]

[0006] The present invention relates to a system and method for incrementally resolving entities using a subset without having to load and analyze the entire identity graph. This incremental approach makes resolution fast enough to function in real time or near real time, even for very large identity graphs. For example, prior art implementations may require hours of resolution for similarly sized identity graphs, whereas some implementations may require less than one second of resolution. This short time frame is important, for example, when data arrives in a continuous stream and entity resolution must be performed within a few hundred milliseconds to keep up with the incoming data stream.

[0007] In applications where the identity graph includes information about consumers, real-time entity resolution using certain embodiments of the present invention can help to fully and accurately convey consumer activity in real time or near real time, thereby enabling the construction of timely consumer segments, which can help improve the performance of digital data processing. Additionally, this can help to perform timely analysis of customer behavior.

[0008] These and other features, objects, and advantages of the present invention will be better understood from the following detailed description of the preferred embodiments and the appended claims, taken in conjunction with the drawings described below. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic diagram illustrating a message bus and streaming process according to one embodiment of the present invention. [Figure 2] FIG. 1 is a schematic diagram illustrating a graph database and a more detailed view of the streaming process, according to one embodiment of the present invention. [Figure 3] FIG. 2 is a data flow diagram of a prospecting process according to one embodiment of the present invention. [Figure 4] FIG. 1 is a diagram of a computer for implementing one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0010] Before describing the present invention in further detail, it is to be understood that the invention is not limited to the particular embodiments described, since the scope of the present invention is limited only by the claims, and that the terminology used in describing particular embodiments is for the purpose of describing those particular embodiments only, and is not intended to be limiting.

[0011] The implementations of the invention described herein rely on three components, as shown in FIGS. 1 and 2. The first is a message bus 10 used to distribute data events. This may be real-time distribution of data events in some embodiments, batch distribution as shown in FIG. 1, or both. The second is a stream process 12 that provides the computing environment for the various components described herein. The stream process 12 may be implemented as an Apache Spark engine, for example, using software from the Apache Spark Foundation in a cloud computing environment. The third component is a graph database 14, shown only in FIG. 2, that stores the nodes and edges of an identity graph used in connection with the implementations, as further described herein. Specifically, the graph database 14 may include pairs (edges) 30, entities (nodes) 32, and predictions (data) 34, as described below.

[0012] 2, the implementation includes a real-time, stream-based data ingestor 16. This component reads records (or data events), parses the records in an event analyzer 18, performs validation in a validator 20, and prepares the records for resolution. The ingestor may further include a classifier 22 that maps the input records to a standard data vocabulary.

[0013] In a stream process, one implementation may include a prediction node and edge selector 24 that receives entities from an entity creation process 26. This element is illustrated in FIG. 1, and FIG. 2 provides a data flow of the operation of the prediction node and edge selector 24. This element identifies portions of the identity graph that may be affected (i.e., changed) by the arrival of new data records / events. This prediction process, illustrated in FIG. 3, reduces the number of comparisons (or matches) that need to be made to perform resolution, thereby enabling improved performance of the described implementation of the present invention. Prediction produces a set of blocks that are matches. Search takes input entity descriptions from a micro-batch 50 (i.e., a reduced section), applies prediction rules (52), and assigns the entity descriptions to one or more blocks based on prediction rules 54. Prediction rules 54 ensure that the resulting block size is a very small subset of the entire identity graph in the graph database 14. At the same time, prediction rules 54 also ensure that if two entity descriptions potentially refer to the same real-world entity, they end up in at least one of the blocks.

[0014] Prediction is the step of gathering all possible matches based on a blocking key 56 that the system generates for each input entity. As an example, in one implementation, an email address may be the blocking key 56, such that all entities with the same email address are grouped together in a grouping, referred to herein as a block. All entities in each block are compared to each other. Thus, for a block of size N, N*N=N 2 comparisons will be made. In that case, we can see that reducing the block size has a dramatic impact on the time required to perform all the comparisons.

[0015] Using prediction rules 54, a set of blocking keys 56 is generated for each event from the micro-batch 50 and maintained in an inverted index database 58. In this database 58, the blocking key (index) 56 references the event to which it belongs. For each new micro-batch 50 of events arriving via bus 10, the system applies the same prediction rules 54 to generate a blocking key 56. The system then performs a search in the inverted index database in a prediction search process 60 to retrieve events with the same blocking key 56. Events in the existing graph retrieved from the graph database 14 form a prediction set in a prediction set generation process 62. For each blocking key from blocking rules 56, the retrieved event is merged with the new event, and the system proceeds to a pairwise match process 64. If the match results in a new edge, the system updates the graph in the graph database 14 with the new edge (66). At the end of this process, the system also updates the inverted index 58 with the new event.

[0016] The particular prediction rule 54 depends on the nature of the data. In various implementations of the present invention, the prediction rule 54 may be manually provided by a data analyst, automatically created by a system, or both. An analyst may combine two columns to create a prediction rule 54. For example, an analyst may combine last name and zip code, or last name and phone number to create a prediction rule 54.

[0017] When prediction rules 54 are created automatically by the system, the following steps may be followed: First, the system creates an ordered string by combining all columns of the event. To generate a signature representing the event, the MinHash algorithm may be applied. MinHash is a technique for quickly estimating how similar two sets are without having to fully compare every element in the two sets. Third, the system applies locality-sensitive hashing to generate a blocking key. Locality-sensitive hashing is a technique for hashing items into buckets with other items that are highly likely to be similar, again without requiring a full comparison.

[0018] In another implementation of the present invention, embeddings can be generated using an ordered string created by combining all columns of the event. These embeddings are vector representations and can be generated using machine learning (ML) models. These embeddings can be used to perform a neighborhood search to create blocks.

[0019] In summary, the pair generator 62 generates potentially matching pairs of records (events). The pairwise matcher 64 then performs a matching function that compares two events / records and emits the probability of a match. The graph upgrader 66 provides for adding new nodes to the identity graph, inserting matched pairs (edges) 30, or deleting existing pairs (edges) 30. The stable ID assigner 68 performs the function of either reusing existing IDs or assigning new IDs. The change log generator generates changes to the identity graph in the graph database 14 due to the resolution process.

[0020] The methods described herein may be implemented by any combination of hardware and software in various embodiments. For example, in one embodiment, a method may be implemented by a computer system (e.g., a computer system such as that of FIG. 4) or a collection of computer systems, each of which includes one or more hardware processors that execute program instructions stored on computer-readable physical storage media coupled to the hardware processors. The program instructions may implement the functions described herein (e.g., the functions of various hardware servers and other components that implement the network-based cloud and non-cloud computing resources described herein). The various methods described herein that are illustrated in the figures represent example implementations. The order of any method may be changed, and various elements may be added, modified, or omitted.

[0021] 4 is a block diagram illustrating an exemplary computer hardware system according to various embodiments. Computer system 500 may implement the hardware portion of a cloud computing system or a non-cloud computing system, forming part of various implementations of the present invention. Computer system 500 may be any of various types of hardware devices, including, but not limited to, a general-purpose server, a personal computer system, a desktop computer, a laptop or notebook computer, a mainframe computer system, a handheld computer, a workstation, a network computer, a consumer device, an application server, a physical storage device, a telephone, a mobile phone, or generally any type of computing node, computing node, computing device, and / or hardware computing device.

[0022] Computer system 500 includes one or more hardware processors 601a, 601b...601n (each processor may include multiple processing cores, which may be single-threaded or multi-threaded), coupled to physical system memory 602 via an input / output (I / O) interface 604. Computer system 500 may further include a network interface 606 coupled to I / O interface 604. In various embodiments, computer system 500 may be a single-processor system including one hardware processor 601a, or a multiprocessor system including multiple hardware processors 601a, 601b...601n, as shown in FIG. 4. Processor 601a, etc. may be any suitable processor capable of executing computational instructions. For example, in various embodiments, processor 601a, etc. may be a general-purpose processor or an embedded processor implementing any of a variety of instruction set architectures. In a multiprocessor system, each of processors 601a, etc. may generally, but need not necessarily, implement the same instruction set. Computer system 500 also includes one or more hardware network communication devices (e.g., network interface 606) for communicating with other systems and / or components over a communications network, such as a local area network, a wide area network, or the Internet. For example, a client application executing on system 500 may use network interface 606 to communicate with a server application executing on a single hardware server or a cluster of hardware servers that implements one or more of the components of the systems described herein in a cloud computing environment or in a non-cloud computing environment, such as those implemented in various subsystems.In another example, an instance of a server application running on computer system 500 may use network interface 606 to communicate with other instances of the application that may be running on other computer systems.

[0023] In the illustrated embodiment, computer system 500 also includes one or more physical persistent storage devices 608 and / or one or more I / O devices 610. In various embodiments, persistent storage device 608 may correspond to a disk drive, a tape drive, a solid-state memory or drive, other mass storage device, or any other persistent storage device. Computer system 500 (or a distributed application or operating system running on computer system 500) may store instructions and / or data in persistent storage device 608 as desired and retrieve the stored instructions and / or data as needed. For example, in some embodiments, computer system 500 may implement one or more nodes of a control plane or control system, and persistent storage 608 may include solid-state drives (SSDs) attached to the server nodes. Multiple computer systems 500 may share the same persistent storage device 608 or may share a pool of persistent storage devices, where the devices in the pool represent the same or different storage technologies, including those described above.

[0024] The computer system 500 includes one or more physical system memories 602 capable of storing code / instructions 603 and data 605 accessible by the processor 601a or other processors. The system memory 602 may include multiple levels of memory and memory caches within the system, designed to swap information within the memory based on access speed, for example. Interleaving and swapping can extend to persistent storage devices 608 in virtual memory implementations where memory space is mapped to persistent storage devices 608. Technologies used to implement the system memory 602 may include, by way of example, static random-access memory (RAM), dynamic RAM, read-only memory (ROM), non-volatile memory, solid-state memory, or flash memory. Similar to persistent storage devices 608, multiple computer systems 500 may share the same system memory 602 or may share a pool of system memories 602. One or more system memories 602 may contain program instructions 603 executable by a processor 601a or the like to implement the routines described herein.

[0025] In various embodiments, program instructions 603 may be coded in binary, assembly language, any interpreted language such as Java, a compiled language such as C / C++, or any combination thereof, although the particular languages ​​shown herein are merely examples. In some embodiments, program instructions 603 may implement multiple separate client, server nodes, and / or other components.

[0026] In some implementations, program instructions 603 may include executable instructions to implement an operating system (not shown), which may be any of a variety of operating systems, such as UNIX, LINUX, Solaris, MacOS, or Microsoft Windows. Any or all of program instructions 603 may be provided as a computer program product or software, which may include a non-transitory computer-readable storage medium having stored thereon instructions that can be used to program a computer system (or other electronic device) to perform a process according to various implementations. A non-transitory computer-readable storage medium may include any mechanism for storing information in a form (e.g., software or processing application) readable by a machine (e.g., a physical computer). Generally speaking, a non-transitory computer-accessible medium may include a computer-readable storage medium or memory medium, such as a magnetic or optical medium, e.g., a disk or DVD / CD-ROM, coupled to or in communication with computer system 500 via I / O interface 604. Non-transitory computer-readable storage media may also include any volatile or non-volatile media, such as RAM or ROM, which may be included as system memory 602 or another type of memory in some embodiments of computer system 500. In other implementations, program instructions may be communicated using optical, acoustic, or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.) conveyed over a communications medium, such as a network and / or a wired or wireless link, as may be implemented via network interface 606. Network interface 606 may be used to interface with other devices 612, which may include other computer systems or any type of external electronic device.

[0027] In some embodiments, system memory 602 may include a data store 605 as described herein. In general, system memory 602 and persistent storage 608 may be accessible on other devices 612 over a network and may store data blocks, replicas of data blocks, metadata related to the data blocks and / or the state of the data blocks, database configuration information, and / or any other information usable in performing the routines described herein.

[0028] In one embodiment, I / O interface 604 may coordinate I / O traffic between, for example, processor 601a, system memory 602, and any peripheral devices in the system, including those via network interface 606 or other peripheral interfaces. In some embodiments, I / O interface 604 may perform any necessary protocol conversion, timing conversion, or other data conversion to convert data signals from one component (e.g., system memory 602) into a format suitable for use by another component (e.g., processor 601a). In some embodiments, I / O interface 604 may include support for devices attached via various types of peripheral buses, such as, for example, variants of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard. Also, in some embodiments, some or all of the functionality of I / O interface 604, such as the interface to system memory 602, may be incorporated directly into processor 601a.

[0029] The network interface 606 may enable data exchange between the computer system 500 and other devices attached to the network, such as, for example, other computer systems (which may implement server nodes, primary nodes, read-only nodes of one or more storage systems, and / or clients of the database systems described herein). Additionally, the I / O interface 604 may enable communication between the computer system 500 and various I / O devices 610 and / or remote storage devices 608. In some embodiments, the input / output devices 610 may include one or more display terminals, keyboards, keypads, touchpads, scanning devices, voice or optical recognition devices, or any other devices suitable for inputting or retrieving data by one or more computer systems 500. The input / output devices 610 may connect directly to a particular computer system 500 or may generally connect to multiple computer systems 500 in a cloud computing environment, a grid computing environment, or other system, including multiple computer systems 500. Multiple input / output devices 610 may be present in communication with computer system 500 or may be distributed among various nodes of a distributed system that includes computer system 500. In some embodiments, similar input / output devices may be separate from computer system 500 and may interact with one or more nodes of a distributed system that includes computer system 500 via a wired or wireless connection, such as network interface 606. Network interface 606 may generally support one or more wireless networking protocols (e.g., Wi-Fi / IEEE 802.11, or another wireless networking standard). Network interface 606 may support communication over any suitable wired or wireless general-purpose data network, such as, for example, other types of Ethernet networks.Additionally, network interface 606 may support communications over a telecommunications / telecommunications network, such as an analog voice network or a digital fiber communications network, a storage area network, such as a Fibre Channel SAN, or any other suitable type of network and / or protocol. In various embodiments, computer system 500 may include more, fewer, or different components (e.g., a display, a video card, an audio card, peripheral devices, or an Ethernet interface) than those shown in FIG.

[0030] Any of the embodiments of the distributed system described herein, or any of the components of the embodiments, may be implemented as one or more network-based services in a cloud computing environment. For example, read-write and / or read-only nodes in a database hierarchy of a hardware database system may present database services and / or other types of physical data storage services using the distributed storage system described herein to clients as network-based services. In some embodiments, a network-based service may be implemented by software and / or hardware systems designed to support interoperable machine-to-machine interaction over a network. A web service may have an interface described in a machine-processable format. Other systems may interact with the network-based service in a manner specified by the network-based service's interface description. For example, a network-based service may define various operations that other systems can invoke and may define specific application programming interfaces (APIs) to which other systems may be expected to conform when requesting the various operations.

[0031] In various embodiments, a network-based service may be requested or invoked by using a message that includes parameters and / or data associated with the network-based service request. Such messages may be formatted according to a particular markup language, such as Extensible Markup Language (XML), and / or may be encapsulated using a protocol. To perform a network-based service request, a client of the network-based service may assemble a message that includes the request and communicate the message to an addressable endpoint (e.g., a Uniform Resource Locator (URL)) corresponding to the web service using an Internet-based application-layer transport protocol, such as Hypertext Transfer Protocol (HTTP).

[0032] In some embodiments, a network-based service may be implemented using Representational State Transfer (REST) ​​techniques rather than message-based techniques. For example, a network-based service implemented according to REST techniques may be invoked by parameters contained within an HTTP method such as PUT, GET, or DELETE.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Although any methods and materials similar or equivalent to those described herein can also be used in the practice or testing of the present invention, a limited number of exemplary methods and materials are described herein. It will be apparent to those skilled in the art that many more modifications are possible without departing from the concept of the present invention.

[0034] All terms used herein should be interpreted as broadly as possible within the context. In particular, the terms "comprises" and "comprising" refer to elements, components, or steps in a non-exclusive manner and should be interpreted to indicate that the referenced element, component, or step may be present, utilized, or combined with other elements, components, or steps not expressly mentioned. When groupings are used herein, all individual elements of the group and all possible combinations and subcombinations of the group are intended to be included individually. When ranges are given herein, the ranges are intended to include all subranges and all individual points within the ranges. When terms such as "about," "approximately," and the like are used herein, they are intended to include amounts, measurements, etc. that do not deviate significantly from the explicitly stated amounts, measurements, etc., so as not to lose sight of the purpose of the device or process being described. All references cited herein are incorporated by reference to the extent they do not contradict the disclosure herein.

[0035] The present invention has been described with reference to certain preferred and alternative embodiments that are intended to be illustrative only and not limiting of the full scope of the invention as set forth in the appended claims.

Claims

1. A method for solving the problem, Distributing a data stream on a message bus, the data stream comprising a plurality of data events; reading and analyzing the plurality of data events; creating a data entity from at least one of the plurality of data events; applying a set of prediction rules in an identity graph in a graph database to identify blocks of the identity graph that are likely to be modified by the entry of the data entity into the identity graph; performing resolution of the data entity by matching the data entity with the block of the identity graph; A method comprising:

2. The method of claim 1 , wherein the plurality of data events are delivered in real time on the message bus.

3. The method of claim 1 , wherein the plurality of data events are delivered in a batch mode on the message bus.

4. The method of claim 1 , further comprising the steps of validating the plurality of data events and mapping the plurality of data events to a standard data vocabulary.

5. 2. The method of claim 1 , wherein the step of identifying blocks of the identity graph that may be modified by entries of the data entity comprises applying a blocking key from a set of blocking keys.

6. 6. The method of claim 5, further comprising the steps of: storing the set of blocking keys in an inverted index database; and performing a search in the inverted index database to identify each data entity having the same blocking key.

7. The method of claim 6 , further comprising generating a set of predictions, performing pairwise matching, and then updating the identity graph with any new edges resulting from the pairwise matching.

8. The method of claim 7 , further comprising ensuring that any two entity descriptions that refer to the same real-world entity are assigned to the same block.

9. a message bus configured to deliver a plurality of data events; a graph database containing an identity graph; a stream process in communication with the message bus and the graph database; 1. A computerized solution system comprising: said stream process comprising one or more computer processors and a memory space in which instructions are stored, said instructions, when executed by said one or more computer processors, causing said one or more computer processors to: reading and analyzing said plurality of data events; creating a data entity from at least one of the plurality of data events; applying a set of prediction rules to identify blocks of the identity graph that are likely to be modified by the entry of the data entity into the identity graph; and performing resolution of the data entity by matching the data entity with the block of the identity graph; A computerized solution system that allows

10. The system of claim 9 , wherein the message bus is configured to perform real-time delivery of the plurality of data events.

11. The system of claim 9 , wherein the message bus is configured to perform batch delivery of the plurality of data events.

12. 10. The system of claim 9, wherein the instructions, when executed by the one or more computer processors, further cause the one or more computer processors to validate the plurality of data events and map each of the plurality of data events to a standard data vocabulary.

13. 10. The system of claim 9, wherein the instructions, when executed by the one or more computer processors, further cause the one or more computer processors to apply a blocking key to the block from a set of blocking keys.

14. 14. The system of claim 13, further comprising an inverted index database for storing the set of blocking keys, the inverted index database configured to perform a search to identify each data entity having the same blocking key.

15. 15. The system of claim 14, wherein the instructions, when executed by the one or more computer processors, further cause the one or more computer processors to generate a prediction set, perform pairwise matching, and then update the identity graph with any new edges resulting from the pairwise matching.

16. 14. The method of claim 13, wherein the instructions, when executed by the one or more computer processors, cause the one or more computer processors to ensure that any two entity descriptions that refer to the same real-world entity are allocated to the same block.

17. When executed, the computer receiving a data stream from a message bus, the data stream comprising a plurality of data events; reading and analyzing said plurality of data events; creating a data entity from at least one of the plurality of data events; applying a set of prediction rules in an identity graph in a graph database to identify blocks of the identity graph that are likely to be modified by entry of the data entity into the identity graph; and performing resolution of the data entity by matching the data entity with the block of the identity graph; A machine-readable, non-transitory physical medium that stores machine-readable instructions that cause a

18. 20. The machine-readable non-transitory physical medium of claim 17, wherein the stored machine-readable instructions, when executed, cause a computer to apply a blocking key from a set of blocking keys to the block.

19. 20. The machine-readable non-transitory physical medium of claim 18, wherein the stored machine-readable instructions, when executed, further cause a computer to store the set of blocking keys in an inverted index database and perform a lookup within the inverted index database to identify each data entity having the same blocking key.

20. 20. The machine-readable non-transitory physical medium of claim 19, wherein the stored machine-readable instructions, when executed, further cause a computer to generate a prediction set, perform pairwise matching, and then update the identity graph with any new edges resulting from the pairwise matching.

Images