Systems and methods for matching within complex data sets

Abstract

A system and method for efficient matching within extremely large data sets uses two data stores. The search data store contains an identifier corresponding to a localized portion of the primary data store, along with a match value for that portion of the primary data store. To perform a match operation, the match value is first searched for from the search data store. The identifier for the localized portion of the primary data store is then retrieved from the search data store. This identifier can be used to perform a search only on the corresponding localized portion of the primary data store, rather than on the entire primary data store.

Description

[Technical Field]

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 085,637, entitled "System and Method for Matching into a Complex Dataset," filed September 30, 2020. Such application is incorporated herein by reference in its entirety. [Background technology]

[0002] Data about objects can be stored in a graph data structure, or data graph. In a data graph, the nodes of the graph are data elements, and the edges are relationships between the data elements. An identity graph is a special type of data graph used to store and access data about natural persons, their families, or their occupations. However, data graphs can also be used to store information about any type of object. Edges in a data graph are used to link together data about the same object. For example, in an identity graph, nodes can contain various contact points associated with the same object, such as a name, postal address, email address, phone number, etc. There may also be a "primary" node for the natural person or occupation, as well as family nodes, that link to each associated contact point. Using a graph data structure, all contact points associated with the same object can be easily identified by following the edges connecting the contact point nodes. This structure therefore allows for a comprehensive understanding of the individual objects for which data is maintained in a way that makes that data easily and quickly accessible.

[0003] In a typical commercial identity graph maintained by a commercial data service provider, a significant amount of the data contained within the graph is purchased or contracted from multiple data vendors. Therefore, the data in the identity graph is frequently updated or expanded as new or updated data becomes available. However, given the enormous size of these data structures, updates are computationally expensive. To add or update data into a data graph, the data service provider must first determine whether such data exists and find a matching data item in the existing graph. With data graphs traversed by millions or even billions of nodes and trillions of edges, what appears to be a simple match search can quickly become computationally infeasible. This is especially true when a new data set matched against the identity graph may itself contain data for millions of objects. Simply walking through the entire data set and comparing individual records with records pulled from the data graph cannot be performed for a length of time that would yield meaningful results, and the data may often already be out of date by the time such an update is completed. In any event, the cost of performing this type of update appears to make the process impractical.

[0004] One approach to speeding up this matching process is to develop specialized lookup indexes, such as those built using the Apache Lucene open-source text search engine library. In certain embodiments, the developed Lucene can be made available through an application programming interface (API) service, which can then be called from a remote location to perform the search and matching process. While this improves the speed of full data graph searches, it is still relatively slow and becomes infeasible when the number of new data sets and the size of those data sets exceed a certain threshold. Therefore, improved data structure systems and methods that provide feasible high-speed matching for extremely large data sets of objects, such as, but not limited to, data graphs, and in particular identity graphs, are desirable.

[0005] The references mentioned in this Background section are not admitted to be prior art with respect to the present invention. Summary of the Invention

[0006] The present invention is directed to a system and method for efficiently matching within extremely large data sets. Two data stores are used. The search data store contains an identifier corresponding to a localized portion of the primary data store, along with a match value for that portion of the primary data store. To perform a match operation, the match value is first searched for in the search data store. Once a match value is found, an identifier for the localized portion of the primary data store is retrieved from the search data store. This identifier can be used to perform a search only on the corresponding localized portion of the primary data store, rather than on the entire primary data store. While this approach is counterintuitive in that two searches must be performed instead of a single search, the extremely large size of the primary data store makes this method much more effective than a single search.

[0007] The present invention may be used in connection with data graphs, particularly identity graphs. However, it will be appreciated that the invention is not so limited and, in fact, may be used in connection with any type of data storage device whose structure can essentially be divided into localized portions. Non-limiting examples of such data structures include trees, maps, lists, and queues.

[0008] These and other features, objects and advantages of the present invention will be better understood from consideration of the following detailed description of the preferred embodiment, together with the drawings and description, which are intended to be limited only by the claims which follow in the application. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 illustrates a high-level data flow architecture for an embodiment of the present invention. [Figure 2] FIG. 1 illustrates a detailed data flow architecture for an embodiment of the present invention, along with bursting of transactions or records into multiple data points. [Figure 3] FIG. 1 illustrates a high-level hardware architecture for an embodiment of the present invention. [Figure 4] FIG. 1 is a swim lane diagram for an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0010] Before describing the present invention in further detail, it is to be understood that the invention is not limited to the particular embodiments described, and that the terminology used in describing the particular embodiments is merely for the purpose of describing those particular embodiments and is not intended to be limiting, since the scope of the present invention will be limited only by the claims.

[0011] Figure 1 illustrates a high-level process flow for the present invention, which in this example is implemented as two data stores: a value-location database 10 and a composite dataset database 12. The value-location database 10 maps values ​​to specific, localized portions of the larger composite dataset database 12. The value-location database 10 uses values ​​to look up identifiers, which are associated with some segments of the composite dataset. The composite dataset database 12 is used to parse identifiers from the value-location database 10 into the data underlying the composite dataset segments in the composite dataset database 12.

[0012] Following the flow portion of FIG. 1 , in receive record step 14, a record having transactions or data is received. For example, this may be a record for an object that requires matching in the composite dataset database. The record may include multiple fields, each of which contains a data item related to the object the record relates to. For example, the object may be a consumer, and fields may include a postal address, an email address, a telephone number, etc. Matching a record may be necessary, by way of non-limiting example, to add data to the composite dataset database 12, to verify data in the composite dataset database 12 or a record, or to retrieve data from the composite dataset database 12 that corresponds to data in the record. The first process then uses values ​​in the transaction or record in find identifier step 16 to find an identifier associated with a portion of the composite dataset in the composite dataset database 12. Once the identifier is found, the corresponding portion of the composite dataset is then searched for a match to the transaction or record in match search step 18. Because the entire complex dataset does not need to be searched, operations proceed more quickly when presented with extremely large datasets. This method can achieve greater efficiency over a traditional one-step search of the entire complex dataset in the complex dataset database 12 as the size of the complex dataset database increases.

[0013] FIG. 2 provides more detail than FIG. 1 and illustrates how, in certain embodiments, the utility of the present invention can be enhanced by “bursting” incoming transactions or records. Given a record with a transaction (TX) or data to be matched in receive record step 14, the process decomposes the transaction and bursts it into multiple data points in burst step 20. Next, to find a domain identifier that matches the data point, each data point in the TX database point set 22 is matched against the value-location database 10 in find identifier step 16. Matching proceeds according to a set of matching rules 24. Matching may not be limited to exact matching in certain instances. For example, matching can proceed through a series of matching rules from match rule 24, starting with an exact match, and then cascading Levenshtein distance and other “fuzzy” matching rules. This results in a known localized domain identifier 26, which is output by the matching process in find identifier step 16. These identifiers 26 are then used to look up all contextual data from the composite data set database 12 in a match search step 18. This provides all contextual data for the originally submitted transaction. The output result is the match data (output data and context) 52 desired by the operation.

[0014] FIG. 4 shows a swim-lane diagram of the process of FIG. 3 , but includes a build process 40 for generating the value-location database 10 and the composite data set database 12. The process begins with a composite data set 42, which is the original data set before it is decomposed into the value-location database 10 and the composite data set database 12. The build process starts with the composite data set 42 and outputs two sets of data. The first is a domain identifier 44 that can be used to match back to the original data set. The second is a lookup table 46 that can be used to find a domain identifier for each data point in the domain identifier 44. The matching service of FIG. 3 is the functionality provided by the burst step 20, find identifier step 16, and match search step 18, which are described in detail above. Similarly, the matching rules 24 are used as described above. In certain embodiments, there is a real-time process 28 that operates in conjunction with the batch mode operation described herein, and as input to the real-time process 28, match requests can be performed in real time as needed using new data 50. Regardless of whether the process is running in real time or batch mode, the results of the match request are returned in output data and context 52.

[0015] Figure 3 illustrates the minimum hardware topology required to operate this system at a high level. The flows in Figures 1 and 2 utilize bursting, lookup, and context processes within the context of a Hadoop Cluster, as shown in Figure 3. The various constructed data sets reside within the context of BLOB storage 30, while the entire composite data set 42 resides within database 32. The Hadoop cluster 34 shown in Figure 3 provides a cluster that, at a minimum, requires at least one master server 36 and at least one worker server 38. This hardware structure is configured to handle large data sets, and its flexible design allows it to scale to any desired size. This means that while such a minimal architecture will work to complete processes on small data sets, at larger data scales, the number of workers 38 and the hardware requirements of these workers 38 will increase.

[0016] In one example, the default hardware requirements for processing 150 TB of data in a complex dataset are as follows: one master node 36 with 8 CPU cores, 52 GB main memory, and 512 GB disk space; 512 worker nodes 38 with 120 GB main memory per worker and 1200 GB total disk space required, with 32 cores per worker node allocated as 2 x 375 GB local solid-state drive (SSD) storage per worker and 450 GB local hard drive (HDD) storage per worker, for a total of 16,384 CPU cores required. Because these are defaults, they can be reduced or increased to accommodate specific datasets, sizes, and scales. In certain embodiments, system requirements can be dynamically scaled up or down in response to load on the system.

[0017] In one example, the system can be implemented within the Google Cloud framework. In this case, the blob store 30 would be Google Cloud Storage, and the Apache Hadoop cluster shown in FIG. 3 would be implemented on Google Cloud Platform Dataproc. Database 32 communication with the Apache Hadoop Cluster can be implemented as Google Cloud BigTable. Many other implementations are possible, so long as there is storage media for the two data stores (physically or virtually separated) and the ability to query processes running on the data stores. Examples include Google BigQuery, MySQL, Postgres, Microsoft SQL Server, RocksDB, LevelDB, BadgerDB, Snowflake, Cassandra, Google Data Store, Mongo, AWS RDS, AWS DynamoDB, Apache Hive, Google Cloud Storage, and AWS S3. The system in various embodiments is not limited to a particular software implementation language; specific examples that can be used for implementation of the present invention include Apache Hadoop, Apache Beam, Apache Spark, Apache Hive, Google BigQuery, Java, Python, Perl, C, Scala, and Apache PIG.

[0018] More generally, the systems and methods described herein can be implemented by any combination of hardware and software in various embodiments. For example, in one embodiment, the systems and methods can be implemented by a computer system or collection of computer systems, each of which includes one or more processor-executed program instructions stored on a computer-readable storage medium coupled to the processor. The program instructions can implement the functionality described herein. The various systems and displays shown in the figures and described herein represent example embodiments. The order of any method may be changed, and various elements may be added, modified, or omitted.

[0019] The computing systems or computing devices described herein may embody the hardware portion of a cloud computing system or a non-cloud computing system, forming part of various embodiments of the present invention. The computing system may be any of a variety of types of devices, including, but not limited to, a commodity server, a personal computer system, a desktop computer, a laptop or notebook computer, a mainframe computer system, a handheld computer, a workstation, a network computer, a consumer device, an application server, a storage device, a telephone, a mobile telephone, another mobile computing device, or generally any type of computing node, compute node, compute device, and / or computing device. The computing system includes one or more processors (any of which may include multiple processing cores, which may be single-threaded or multi-threaded) coupled to system memory via an input / output (I / O) interface. The computing system may further include a network interface coupled to the I / O interface.

[0020] In various embodiments, the computer system may be a single-processor system including one processor or a multiprocessor system including multiple processors. The processor may be any suitable processor capable of executing computational instructions. For example, in various embodiments, the processor may be a general-purpose processor or an embedded processor implementing any of a variety of instruction set architectures. In a multiprocessor system, each of the multiple processors typically, but not necessarily, implements the same instruction set. The computer system also includes one or more network communication devices (e.g., network interfaces) for communicating with other systems and / or components over a communications network, such as a local area network, a wide area network, or the Internet. For example, a client application executing on the computing device may use the network interface to communicate with a server application executing on a single server or a cluster of servers, where one or more of the components of the systems described herein are implemented in a cloud computing environment or a non-cloud computing environment, as implemented in various subsystems. In another example, an instance of a server application executing on a computer system may use the network interface to communicate with other instances of the application, which may be implemented on other computer systems.

[0021] The computing device also includes one or more persistent storage devices and / or one or more I / O devices. In various embodiments, the persistent storage device may correspond to a disk drive, a tape drive, solid-state memory, other mass storage device, or any other persistent storage device. A computer system (or a distributed application, or an operating system on a distributed application) may store instructions and / or data in the persistent storage device as needed and retrieve the stored instructions and / or data as needed. For example, in some embodiments, a computer system may implement one or more nodes of a control plane or control system, and the persistent storage device may include SSDs attached to the server nodes. Multiple computer systems may share the same persistent storage device, or may share a pool of persistent storage devices, with devices in the pool representing the same or different storage technologies.

[0022] A computer system includes one or more system memories capable of storing code / instructions and data accessible by a processor. System memory can include multiple levels of memory and memory caches, for example, in systems designed to swap information within memory based on access speed. Interleaving and swapping can be extended to persistent storage in virtual memory implementations. Technologies used to implement memory can illustratively include static random access memory (RAM), dynamic RAM, read-only memory (ROM), nonvolatile memory, or flash-type memory. As with persistent storage, multiple computer systems can share the same system memory or share a pool of system memory. One or more system memories can contain program instructions executable by a processor to implement the routines described herein. In various embodiments, the program instructions can be encoded in binary, Assembly language, any interpreted language such as Java, a compiled language such as C / C++, or any combination thereof; the specific languages ​​given herein are merely illustrative. In some embodiments, the program instructions can implement multiple individual client, server nodes, and / or other components.

[0023] In some embodiments, the program instructions may include instructions that can be executed to implement an operating system (not shown), which may be any of a variety of operating systems, such as UNIX, LINUX, Solaris™, MacOS™, or Microsoft Windows™, or a mobile computing device operating system such as iOS™. Any or all of the program instructions may be provided as a computer program product, i.e., software, which may include a non-transitory computer-readable storage medium having instructions stored thereon, which software may be used to program a computer system (or other electronic device) to thereby perform processes according to various embodiments. A non-transitory computer-readable storage medium may include any mechanism for storing information in a form (e.g., software, processing application) readable by a machine (e.g., a computer). Generally speaking, a non-transitory computer-accessible medium may include a computer-readable storage medium or memory medium, such as a magnetic or optical medium, e.g., a disk or DVD / CD-ROM coupled to a computer system via an I / O interface. Additionally, non-transitory computer-readable storage media may include any volatile or non-volatile media, such as RAM or ROM, that may be included in some embodiments of a computer system as system memory or another type of memory. In other embodiments, program instructions may be communicated using optical, acoustic, or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.) carried over a communications medium, such as a network and / or wired or wireless link, as may be implemented via a network interface. The network interface may be used to interface with other devices, which may include other computer systems or any type of external electronic device.Generally, system memory, persistent storage, and / or remote storage accessible on other devices over a network may store data blocks, replicas of data blocks, metadata associated with the data blocks and / or their state, database configuration information, and / or any other information that can be used to implement the routines described herein.

[0024] In certain embodiments, the I / O interface may coordinate I / O traffic between the processor, system memory, and any peripheral devices in the system, including via a network interface or other peripheral interface. In some embodiments, the I / O interface may perform any necessary protocol, timing, or other data conversion to convert data signals from one component (e.g., system memory) into a format suitable for use by another component (e.g., processor). In some embodiments, the I / O interface may include support for devices attached via various types of peripheral buses, such as variants of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard. Also, in some embodiments, some or all of the functionality of the I / O interface, such as the interface to system memory, may be incorporated directly into the processor.

[0025] A network interface can enable data exchange between a computer system and other devices attached to a network, such as other computer systems (which may embody one or more storage system server nodes, master nodes, read-only nodes, and / or clients of the database systems described herein). Additionally, an I / O interface can enable communication between a computer system and various I / O devices and / or remote storage devices. Input / output devices, in some embodiments, include one or more display terminals, keyboards, keypads, touchpads, scanning devices, voice or optical recognition devices, or any other devices suitable for entering or retrieving data by one or more computer systems. These can be directly connected to a particular computer system or can be collectively connected to multiple computer systems in a cloud computing environment, a grid computing environment, or other system including multiple computer systems. Multiple input / output devices can be provided for communication with a computer system or can be distributed across various nodes of a distributed system including computer systems. A user can view the user interfaces described herein using various types of display screens, which can include CRT displays, LCD displays, LED displays, and other display technologies. In some embodiments, input can be received through a display using touchscreen technology, and in other embodiments, input can be received through a keyboard, mouse, touchpad, or other input technology, or any combination of these technologies.

[0026] In some embodiments, similar input / output devices may be separate from the computer system and may interact with one or more nodes of a distributed system that includes the computer system via a wired or wireless connection, such as via a network interface. The network interface may typically support one or more wireless networking protocols (e.g., Wi-Fi / IEEE 802.11 or another wireless networking standard). The network interface may support communication over any suitable wired or wireless general-purpose data network, such as other types of Ethernet networks. Additionally, the network interface may support communication over telecommunications / telephony networks, such as analog voice networks or digital fiber communication networks, over storage area networks, such as Fibre Channel SANs, or over any other suitable type of network and / or protocol.

[0027] Any of the distributed system embodiments described herein, or any of their components, can be implemented as one or more network-based services in a cloud computing environment. For example, read-write and / or read-only nodes in the database tier of a database system can provide database and / or other types of data storage services to clients as network-based services using the distributed storage system described herein. In some embodiments, a network-based service can be implemented by software and / or hardware systems designed to support interactive machine-to-machine interaction over a network. A web service can have an interface described in a machine-processable format, such as Web Services Description Language (WSDL). Other systems can interact with the network-based service in a manner specified by the network-based service's interface description. For example, a network-based service can define various operations that other systems can provide and can define specific application programming interfaces (APIs) that other systems can expect to validate when requesting various operations.

[0028] In various embodiments, network-based services may be requested or provided using messages that include parameters and / or data associated with the network-based service request. Such messages may be formatted according to a particular markup language, such as Extensible Markup Language (XML), and / or may be encapsulated using a protocol, such as Simple Object Access Protocol (SOAP). To perform a network-based service request, a network-based service client may assemble a message containing the request and communicate the message to an addressable endpoint (e.g., a Uniform Resource Locator (URL)) corresponding to the web service using an Internet-based application-layer transport protocol, such as Hypertext Transfer Protocol (HTTP). In some embodiments, network-based services may be implemented using Representational State Transfer (REST) ​​techniques rather than message-based techniques. For example, network-based services implemented according to REST techniques may be provided via parameters included within HTTP methods such as PUT, GET, or DELETE.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Although any methods and materials similar or equivalent to those described herein can also be used in the practice or testing of the present invention, a limited number of exemplary methods and materials are described herein. It will be apparent to those skilled in the art that many more modifications are possible without departing from the inventive concepts herein.

[0030] All terms used herein should be interpreted in the broadest possible manner consistent with the context. When groupings are used herein, all individual elements of the group and all possible combinations and subcombinations of the group are intended to be individually included in the disclosure. All references cited herein are incorporated by reference to the extent that there is no inconsistency with the disclosure herein. When ranges are used herein, all points within the range and all subranges within the range are intended to be included in the disclosure.

[0031] The invention has been described above with reference to certain preferred and alternative embodiments that are merely illustrative and are not intended to limit the overall scope of the invention.

Claims

1. 1. A method for matching transactions or records against a composite data set, comprising: receiving, by one or more hardware processors of a computer, said transactions or records, said transactions or records including at least one value; one or more hardware processors of said computer bursting said transactions or records into a plurality of burst values; one or more hardware processors of the computer searching a value-location database for each of the plurality of burst values ​​and returning an identifier for each of the plurality of burst values ​​for which a match is found in the value-location database; one or more hardware processors of the computer identifying a localized region of the composite data set corresponding to the returned identifier; searching, by one or more hardware processors of the computer, for a match with at least one from the localized region of the composite data set; one or more hardware processors of said computer returning matching records for said at least one value; A method comprising:

2. 2. The method of claim 1, wherein said step of searching said localized regions of said composite data set for a match with said at least one value comprises searching each of said burst values ​​from each of said identified localized regions.

3. 3. The method of claim 2, wherein returning a matching record for the at least one value comprises returning a plurality of records, each of the plurality of records matching one of the plurality of burst values.

4. 2. The method of claim 1, wherein the step of looking up the value from a value-location database performs an exact match.

5. The method of claim 1 , wherein the step of looking up the value from a value-location database performs a fuzzy match.

6. The method of claim 5 , wherein the fuzzy matching is performed using a Levenshtein distance measure.

Images