System and method for high performance data matching

Abstract

The present disclosure relates to systems and methods for high performance data matching. Entities maintain large amounts of data and desire to reconcile duplicate records. One way to address this problem is through data matching. However, standard data matching at the record level can be laborious and inefficient. To remedy these inefficiencies in data matching, the present disclosure describes a system in which token records are tokenized into token sets again based on the token records satisfying at least one token set rule. The token set rule can be based on the common presence of multiple tokens in a token record. If multiple token records have the required tokens from the set rule, these token records can be hashed and accumulated into a token set (i.e., tokenized into a token set again). Token sets allow for more efficient data matching.

Description

[0001] Cross-reference to related applications

[0002] The present application is related to U.S. patent application Ser. No. 16 / 776,293, entitled “SYSTEMS AND METHOD OF CONTEXTUAL DATA MASKING FOR PRIVATE AND SECURE DATA LINKAGE”; U.S. patent application Ser. No. 17 / 103,751, entitled “SYSTEMS AND METHODS FOR UNIVERSAL REFERENCE SOURCE CREATION AND ACCURATE SECURE MATCHING”; U.S. patent application Ser. No. 17 / 103,720, entitled “SYSTEMS AND METHODS FOR DATA ENRICHMENT”; and U.S. patent application Ser. No. 17 / 219,340, entitled “SYSTEMS AND METHODS FOR AN ON-DEMAND, SECURE, AND PROactive VALUE-ADDED DATA MARKETPLACE”. ON-DEMAND, SECURE, AND PREDICTIVE VALUE-ADDED DATA MARKETPLACE)" which are hereby incorporated by reference in their entirety. Technical Field

[0003] The present disclosure relates to data matching technology. Background Art

[0004] Entities maintain large amounts of data that may be disorganized and / or incomplete. For example, an entity may maintain more than one incomplete record related to a subject such as an individual, a product, an organization, etc. One record may contain the subject's address, email address, gender, and geographic location, while another record may contain the subject's name, address, phone number, date of birth, and credit card information. Each of these records may be incomplete. Similar incomplete records may exist for products and organizations. Currently, entities that wish to reconcile these disorganized records typically must manually combine the records, which results in an inefficient and time-consuming process and the potential exposure of personally identifiable information.

[0005] Due to the breadth of data sources, linking, deduplicating, or finding matches between data assets remains one of the biggest issues for most organizations. Multi-channel data collection and internal replication to meet the needs of customers and business stakeholders presents a reconciliation challenge. Other issues, such as incompleteness, data entry errors, and changes over time make it difficult to establish a master (i.e., "golden") record of data.

[0006] Another problem that entities face is ensuring that the records they can have have data integrity. For example, an entity can have two incomplete records that appear to relate to the same data object. However, one record may list a different email address or phone number than the other record. Such data discrepancies reduce the integrity of the data record and make it more difficult for an entity to reconcile multiple incomplete data records because the entity will be unsure which record is truly correct. Furthermore, the entity may be unsure to what extent a particular record can be correct. Modern enterprises are hampered in terms of accurate data collection and reconciliation.

[0007] Furthermore, data matching and data linking are computationally expensive when applied to data objects (e.g., people) with multiple features, characteristics, and attributes. The potential complexity of data matching with these data objects is multiplied by the need to process permutations to obtain higher accuracy matching results.

[0008] Therefore, there is an increasing need for systems and methods that can address the challenges of modern data collection and reconciliation, including inefficiencies in matching multiple incomplete records of the same data object, loss of integrity when records are inconsistent, and the potential exposure of personally identifiable information (PII), sensitive business information, and / or any form of confidential information when such attempts to match and reconcile data records are made. There is an increasing need to tokenize attributes and attribute sets to improve data matching efficiency.

[0009] It is with respect to these and other general considerations that the various aspects disclosed herein have been made.In addition, although relatively specific problems may be discussed, it should be understood that the examples should not be limited to solving specific problems identified in the background of this disclosure or elsewhere. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Non-limiting and non-exhaustive examples are described with reference to the following figures.

[0011] Figure 1 An example of a distributed system for high performance data matching as described herein is shown.

[0012] Figure 2 An exemplary input processor for implementing systems and methods for high performance data matching as described herein is shown.

[0013] Figure 3 An exemplary method for matching sets of tokens using high performance data matching techniques is shown.

[0014] Figure 4 An example of a distributed system including a client environment and an integrated platform environment for high-performance data matching is shown.

[0015] Figure 5 Exemplary data sources and tokens are shown.

[0016] Figure 6 Examples of matching token pairs, establishing token set rules, and matching token sets are shown.

[0017] Figure 7 An exemplary architecture for high performance data matching is shown.

[0018] Figure 8 An example of a suitable operating environment is shown in which one or more of the present embodiments may be implemented. DETAILED DESCRIPTION

[0019] Aspects of the present disclosure are described more fully below with reference to the accompanying drawings, which form a part of this document and illustrate specific exemplary aspects. However, different aspects of the present disclosure can be implemented in many different forms and should not be construed as being limited to the aspects described herein; rather, these aspects are provided so that the present disclosure will be thorough and complete and the scope of these aspects will be fully conveyed to those skilled in the art. Various aspects can be practiced as methods, systems, or devices. Therefore, various aspects can take the form of hardware implementations, fully software implementations, or implementations combining software and hardware aspects. Therefore, the following detailed description should not be taken as limiting.

[0020] Embodiments of the present application are directed to enhanced data matching techniques (otherwise referred to throughout this disclosure as "high performance data matching"). Data matching and data linking are computationally expensive processes when data matching is applied to data objects (e.g., people) with multiple features, characteristics, and attributes. Tokenizing these data attributes at the record level is resource intensive.

[0021] To address these shortcomings, the present application discloses a data matching process by which tokenized record pairs are subsequently tokenized into "entity tokens" or "token sets." By aggregating these related token-record pairs into a single token set, data matching can be accelerated by multiple orders of magnitude, allowing billions of data matching resolutions to occur in a fraction of the time required for traditional standard attribute-by-attribute data matching.

[0022] In some exemplary aspects, high-performance data matching can be performed in two steps. The first step is tokenization or "data masking", which is an example of a one-way hash. Data masking can refer to the hashing and / or encryption of client-specific information. Throughout this application, "client" can refer to a client and / or a reference source. In order for an entity to transmit sensitive data (e.g., PII) to a third party (e.g., an integration platform), the data should generally be tokenized. In some examples described herein, client-specific information can first be hashed and encrypted and then uploaded to a secure server. Tokenized information can be obtained from a secure server and analyzed. Tokenized client-specific information (i.e., "client token") can be tokenized using a combination of hashing (one-way compression function) and / or encryption, wherein the hash code can include an additional computer-generated secure hash algorithm (SHA2 512 / 256) salt value (salt). Other hashing and encryption methods may be used, including but not limited to SHA2 348, SHA3, Triple DES, Pisces, Blowfish, Advanced Encryption Standard (AEs), IDEA, MD5, and / or RSA security. When the third-party server receives information transmitted from a client or reference source, the information is in the form of a token. In other words, the raw data from the client and / or reference source (i.e., "client") is hashed and encrypted before being transmitted to the third-party secure server. The integration platform does not receive or process the raw data.

[0023] In some exemplary aspects, a first tag may be included in a list of anonymized tags corresponding to a type of information attributed to a token based on a determined classifier. The tag may provide an anonymous identifier of the type of information represented in the attribute. Tags may be generated based on any attribute and classifier. For example, if the attribute is related to a name, the corresponding tag may be "La1". In these embodiments, only entities that have access to a list of information corresponding to the tag can determine the type of information determined by each tag, thereby anonymizing the data. For a third-party integration platform, due to the tokenization of the data, the true unencrypted attribute or classifier may be unknown. For example, an attribute may refer to the address of a data object and have an "address" tag, but once tokenized, the tag may be a token "La1" of a third-party integration platform. Further description of the process may be found in U.S. patent application Ser. No. 16 / 844,927, which is incorporated herein by reference in its entirety.

[0024] Data integration refers to integrating two or more data records to create a single data record. For example, a data record may include data attributes of name, DOB, gender, and email address. Another data record may include data attributes of name, gender, home address, and telephone number. If the two records are integrated, a single data record may be created that includes the data attributes of name, DOB, gender, email address, home address, and telephone number. Once integrated, the integrated data record may be referred to as a "composite."

[0025] Data composites may be useful for entities that may have multiple data records related to a single data object. For example, a bank may have a data record for a data object's checking account and a data record for a home loan for the same data object. Both of these data records may include overlapping information, but one data record may include certain data attributes that the other data record does not. Therefore, it may be beneficial for an entity to create composites of these data records, resulting in greater efficiency and a higher level of data integrity.

[0026] As described herein, entities can utilize the proposed system and method to match certain tokens using a reference token repository. For example, a client may have two databases with two separate records related to the same entity (e.g., a person, a place, a product, a thing, etc.). The client can submit these records to a data shielding application that generates a private and secure token for each of these records. These tokens can then be transmitted to an integration platform where the tokens are analyzed for matching. In some instances, the tokens can be compared to each other (i.e., AB matching). In other instances, the tokens can be compared to a reference token repository to determine matching records (i.e., passing matches). If the tokens match, one of the tokens (i.e., a unique token) can be stored in a reference token repository for future use. Such a process of storing unique tokens ensures an efficient repository because there will be no duplicate tokens in the repository.

[0027] The systems and methods proposed herein utilize reference sources to validate data records and fill in potential missing data attributes of data records. A reference source is an entity with data used as a reference point. For example, a reference source can be a "trusted" source, such as a credit bureau, a bank, and / or a government entity. However, because "trust" is subjective, for the purposes of this application, a reference source is any source used as a reference for comparison with another source (e.g., a customer source). For example, a customer may have multiple data records for a data object, but some data attributes may be missing. The customer may hash and encrypt the data (i.e., convert the raw data into a token), and then upload the token to a secure server where a third-party integration platform may access the customer token. In addition, a reference source may have a complete and verified record (i.e., a "trusted" record), which is hashed and encrypted (i.e., tokenized) and then uploaded to a secure server managed by an integration platform. A third-party integration platform may access a reference source token. The third-party integration platform may then determine overlapping and / or unique tokens in the customer data (i.e., payload) by matching the customer token with a reference source token located in a reference source token repository.

[0028] In some aspects, a client token and a reference source token may be received by the system. The client token and the reference source token may be analyzed and compared. The results of the analysis and comparison may overlap in the client token and / or matching characteristics via comparison with a reference token repository. Specifically, the third-party integration platform enables determination of duplicate records and integration of data, but the actual integration of the data (e.g., merging two records related to the same entity to create a single record) occurs in the client environment.

[0029] The true basis of how data matching (e.g., between client sources, between client sources and reference sources, etc.) can occur is the focus of this application. In one example, token pairs can be formed, where a token pair is defined as two tokens of the same type (e.g., type name, SSN, address, or any combination of attributes can be a "type"). The tokens can then be associated with set rules, which may require a certain number of matching token pairs to qualify as a record match / entity resolution. Once a set rule is established, every token pair that can satisfy the rule is accumulated into a token set. The token set is then re-tokenized into a single token and used for comparison, rather than individual comparisons of each token pair in the rule set, to resolve the entity and find matching records.

[0030] Accordingly, the present disclosure provides a variety of technical benefits, including, but not limited to: supporting more efficient use of electronic resources for data matching, integration, and supporting data enrichment; providing more efficient storage management because matching, integration, and enrichment of data sets can occur at a single third-party server; reducing potential exposure of sensitive data because all data transmitted to the third-party server is tokenized (i.e., hashed and encrypted); and reducing manual overload on electronic devices because high-performance data matching, integration, and enrichment are configured to match sets of tokens rather than individual tokens that may require significantly more computing resources.

[0031] Figure 1 An example of a distributed system for high performance data matching as described herein is shown. The exemplary system 100 presented is a combination of interdependent components that interact to form an integrated whole for integrating and augmenting data. The components of the system can be hardware components or software implemented on and / or executed by the hardware components of the system. For example, the system 100 includes client devices 102, 104, and 106, local databases 110, 112, and 114, one or more networks 108, and server devices 116, 118, and / or 120.

[0032] The client devices 102, 104, and 106 can be configured to receive and transmit data. For example, the client devices 102, 104, and 106 can contain client-specific data. The client devices can download a third-party tokenization software program that can be applied to the client-specific data via one or more networks 108. The client-specific data can be stored in local databases 110, 112, and 114. Once tokenized, the client-specific data is converted into "tokens", and these tokens can be transmitted to one or more servers 116, 118, and / or 120 via one or more networks 108 and / or satellites 122. One or more servers 116, 118, and / or 120 can be third-party servers owned by the integration platform. In other examples, the client-specific data can be stored in a server (in addition to or in place of the local client device and local database), and can be tokenized and then transmitted from the client server to the third-party server via one or more networks 108 and / or satellites 122.

[0033] In various aspects, client devices (e.g., client devices 102, 104, and 106) may have access to one or more data sets or data sources and / or databases that include client-specific data. In other aspects, client devices 102, 104, and 106 may be equipped to receive broadband and / or satellite signals carrying client-specific tokens or unencrypted data (or a mix of both). The signals and information that client devices 102, 104, and 106 may receive may be transmitted from satellite 122. Satellite 122 may also be configured to communicate with one or more networks 108, in addition to being able to communicate directly with client devices 102, 104, and 106. In some examples, client devices may be mobile phones, laptops, tablets, smart home devices, landlines, and wearable devices (e.g., smart watches), among other devices.

[0034] In some examples, client devices 102, 104, and / or 106 may be configured to run high-performance data matching software on certain data sets. Specifically, one or more client devices 102, 104, and / or 106 may be configured to run a matching preprocessor that can partition files consisting of hashed, masked, and / or encrypted tokens. The partitioned files may be fed to a matching processor that may generate intermediate files during the high-performance data matching process. The output of the matching processor may be a file suitable for consumption by a database loading process. The database may be stored on a local storage device 110, 112, and / or 114 or one or more remote storage devices 116, 118, and / or 120. Individual files may be loaded into a database loader and database-ready files may be saved across databases.

[0035] In some examples, one or more local databases 110, 112, and / or 114 and one or more remote data stores 116, 118, and / or 120 may be PostgreSQL (also known as "postgres") databases in which multiple data sources may be stored. Cross-source matching may occur within one or more PostgreSQL databases, and an entity merge processor may run a recursive database function to merge token entities having similar token attributes.

[0036] The process for tokenizing and transmitting data from both the client side and the reference source side may be similar, as the data may be initially stored locally and subsequently hashed and encrypted on the client device owned by the client and / or owned by the reference source, and once in tokenized form, finally transmitted to a third-party server for analysis, integration, and enrichment, among other actions. In other words, Figure 1Depicted are network topologies that may be used in a client environment and / or a reference source environment (ie, client devices 102, 104, and / or 106 may belong to a client environment in one example and a reference source environment in another example).

[0037] Figure 2 An exemplary input processor for implementing systems and methods for high performance data matching as described herein is shown. Input processor 200 can be embedded in a client device (e.g., client devices 102, 104, and / or 106), a remote network server device (e.g., devices 116, 118, and / or 120), and other devices capable of implementing systems and methods for high performance data matching. The input processing system includes one or more data processors and is capable of executing algorithms, software routines, and / or instructions based on processed data provided by at least one client source and / or reference source. The input processing system can be a factory-installed system or an add-on unit to a particular device. In addition, the input processing system can be a general purpose computer or a specialized, special-purpose computer. There is no limitation on the location of the input processing system relative to a client or remote network server device, etc. According to Figure 2 In the illustrated embodiment, the disclosed system may include a memory 205, one or more processors 210, a communication module 215, a matching module 220, and a token (set) determination module 225. The token (set) determination module 225 may be configured to determine both tokens and token sets (also known as "entity tokens"). Other embodiments of the present technology may include some, all, or none of these modules and components, as well as other modules, applications, data, and / or components. However, some embodiments may combine two or more of these modules and components into a single module and / or associate a portion of the functionality of one or more of these modules with different modules.

[0038] The memory 205 may store instructions for running one or more applications or modules on one or more processors 210. For example, the memory 205 may be used in one or more embodiments to hold all or part of the instructions required to perform the functions of the matching module 220 and / or the token (set) determination module 225 and the communication module 215. In general, the memory 205 may include any device, mechanism, or populated data structure for storing information. According to some embodiments of the present disclosure, the memory 205 may include but is not limited to any type of volatile memory, non-volatile memory, and dynamic memory. For example, the memory 205 may be a random access memory, a memory storage device, an optical memory device, a magnetic medium, a floppy disk, a tape, a hard disk drive, a SIMM, a SDRAM, a RDRAM, a DDR, a RAM, a SODIMM, an EPROM, an EEPROM, an optical disk, a DVD, and / or the like. According to some embodiments, the memory 205 may include one or more disk drives, a flash drive, one or more databases, one or more tables, one or more files, a local cache memory, a processor cache memory, a relational database, a flat database, and / or the like. In addition, a person of ordinary skill in the art will understand that many additional devices and techniques for storing information may be used as the memory 205.

[0039] The communication module 215 is associated with sending / receiving information (e.g., matching by the matching module 220, and / or determining by the token (set) determination module 225), receiving commands via a client device or server device, other client devices, remote network servers, etc. These communications can use any suitable type of technology, such as Bluetooth, WiFi, WiMax, cellular (e.g., 5G), single-hop communication, multi-hop communication, dedicated short-range communication (DSRC), or proprietary communication protocols. In some embodiments, the communication module 215 sends information output by the matching module 220 (e.g., matching tokens) and / or information output by the token (set) determination module 225 (e.g., determining a unique token based on the match to be stored in a universal reference token repository) to the client devices 102, 104, and / or 106, and the memory 205 to be stored for future use. In some examples, the communication module can be constructed on the HTTP protocol by using one or more secure REST servers for RESTful services.

[0040] The matching module 220 is configured to receive at least two sets of data, for example, data set A and data set B. The module 220 may also receive and maintain at least one reference data set (e.g., a reference source data set, a universal reference set, etc.). The matching module 220 may be configured to compare data set A and data set B via a matching algorithm (e.g., AB matching). The module 220 may also be configured to compare data set A with a reference data set and to compare data set B with a reference data set. The comparative analysis may reveal that data set A fully matches the reference data set and data set B fully matches the reference data set. If data set A matches the reference data set and data set B matches the reference data set, then data set A matches data set B via transfer performance. The matching module 220 may also accept a single data record and compare the data record with a data record from at least one data set maintained by the transfer matching module 220. The matching module 220 is also configured to perform transfer matching analysis on tokens. In other words, at least two token sets may be compared with a reference source token set, and if token set A matches the reference token set and token set B matches the reference token set, then token set A matches token set B via transfer performance.

[0041] In yet other examples, the matching module 220 can be configured to match data sets using correlation properties, where token pairs are generated and then aggregated to form token sets. For example, different tokens can be associated with different underlying data attributes (which can be masked). Tokenized records can have overlapping tokens, so once these records are compared to each other and determined to have matching token records, these records can be merged into a single token set. This token set can be used for future matching and processing functions, as opposed to using two different - but similar and overlapping - records.

[0042] The token (set) determination module 225 may be configured to determine a single unique token record that is determined to match another token based on the results of the matching module 220, and to determine a group of tokens that may be accumulated (aggregated) into a token set that is determined to match another token set based on the results of the matching module 220. If a match is indicated from the matching module 220, a copy of the matching token (or token set) may be obtained by the token (set) determination module 225 and stored in a reference token repository such as a universal reference token repository. The token (set) determination module 225 may also be configured to determine tokens that are already stored in the token repository. For example, based on the results of the matching module 220, a unique token may be determined, but not stored in the universal reference token repository, because the unique token already exists, and the universal reference token repository will not accept duplicate tokens or token sets.

[0043] Figure 3An exemplary method for matching a set of tokens using high performance data matching techniques is shown. Each token may include a single attribute or multiple attributes. For example, a single token may refer to a group of three attributes (e.g., name, DOB, email), or in some exemplary aspects, a token may refer to a portion of a single attribute (e.g., the last 4 digits of an SSN). Tokens are not limited to matching one-to-one with attributes. Method 300 begins with steps 302 and 304, which receive tokens (or sets of tokens) from a first source and a second source. For example, tokens may be received by a single client from two different databases (i.e., two different sources). Tokens may or may not be associated with the same entity.

[0044] The customer data may be tokenized at the customer side before transmitting the token at steps 302 and 304. Prior to receiving the customer data, the customer's client device may have used a tokenization software program to hash and encrypt (or "mask") the raw customer data prior to transmission, so that what is actually transmitted to the integration platform is the customer "token." Once tokenized, the customer token may be transmitted and received by the third-party integration platform at step 304. Note that receiving the customer token and receiving the reference source token at step 304 may be performed simultaneously or in reverse order (receiving the reference source token occurs before receiving the customer token).

[0045] The client token may be received by the integration platform and stored on a secure server. After receiving the client token, the method proceeds to step 306 where the tokens may be compared to each other. The comparison may utilize a matching algorithm such as AB matching, deterministic matching, probabilistic matching, and / or a combination of matching algorithms. In other examples, the matching technique may rely on a hybrid technique that uses the probability of matching by weighting.

[0046] Each token record may be compared to other token records to determine if there are overlapping tokens in the token records. Overlapping tokens (i.e., "matched" tokens) from each record may be compared to create matching pairs, and ultimately resolved as matches to a "set rule" library. In another exemplary process, a predetermined set rule may determine whether certain matching tokens may trigger consolidation (tokenization) of multiple tokens into a set token because the same matching token exists in multiple records. Thus, the system may determine that it is more efficient to consolidate these nearly duplicate token records into a single token set, thereby enabling more efficient matching and processing.

[0047] After the comparison step in 306, a token set can be generated at step 308, which is the product of a simple weighted and hashed accumulation of multiple token records. For example, the set rule may stipulate that if a token record includes tokens T1, T2, and T4, the token record should be matched. If record 2 has tokens 1, 2, 3, and 4, but record 3 only has tokens 1, 2, and 4, then token records 2 and 3 can be matched according to the set rule because both records include tokens T1, T2, and T4. By using simple weighted matching, these token records have the same tokens for three token types (T1, T2, and T4). The rules require that these token records be integrated into tokens (sets) (e.g., S1R1=S2R2=S(T1, T2, T4)). The following is about Figure 6 This example is further described.

[0048] After determining a single token match, at least one set token (i.e., the integration of at least two customer token records) may be represented. A set token refers to integrating multiple token records to create a single token record. In other instances, a set token may refer to a set of multiple tokens aggregated together. For example, a customer token may constitute multiple token records associated with the same data object (although the third-party integration platform cannot determine the data object because the information is tokenized on the third-party integration platform). A composite of a data object may be created by integrating multiple token records associated with the same data object. When the composite record is transmitted back to the customer and / or reference source environment and mapped via a unique non-derived ID in the customer and / or reference source environment, the integrated record will include all of the above data attributes: name, DOB, gender, physical address, email address, and age. The end result is a composite record, which is a single record associated with a certain data object and / or entity. Such a composite record is created in the customer environment, so while the third-party integration platform can help determine certain records to be integrated and / or augmented, the actual integration and augmentation occurs in the customer environment before the customer token is transmitted to the integration platform. As used herein, a "data object" may be a person, product, business, entity, etc. The systems and methods described herein are not limited to personal data related only to human data objects, but may also be applied to supply chain data, product data, business entity data, etc.

[0049] After the token set is generated at step 308, a unique token set may be determined at step 310, where the generated token set may be compared to pre-existing tokens in a reference token repository. If a token already exists, the token may not be added to the reference token repository to avoid duplication, but if a token set does not exist, the new token may be appended to the reference token repository at step 312.

[0050] Figure 4An example of a distributed system including a client environment and an integrated platform environment for high-performance data matching is shown. Figure 4 The distributed system in is an exemplary data registry 400. The data registry 400 may include a client environment 402 and an integration platform 404. As shown in the client environment 402, data may be received from multiple client databases (e.g., database source A and / or source B). The data may include multiple data attributes associated with a data object, such as name, address, phone, email, SSN, DOB, gender, etc. The customer data may be tokenized within the client environment 402. Once tokenized, the client token may be transmitted to the integration platform 404.

[0051] Integration platform 404 can receive tokens from customer environment 402 in the form of private and secure tokens. Integration platform 404 can receive tokens from customer environment in the form of private and secure tokens. Within integration platform 406, tokens from customers can be compared, for example, using AB matching techniques. At the comparison step, tokens can be used to determine link pairs (i.e., "overlaps"). In some examples, records can also consist of bit strings (i.e., strings of 1s and 0s). "1" can indicate that a certain data attribute exists, while "0" can indicate that the data attribute does not exist. When comparing the tokens of the records, if one or more tokens of the records have the same value, a link pair is established based on an established threshold. If the bit values ​​in the matching record pair are different, at least one token record in the token record has a data attribute, while the other token record does not. If both token records have "0" in the column, the specific corresponding data attribute is missing. Specifically, a link pair can be created by a customer token transmitted to integration platform 404. The customer token can then be compared with at least one reference source token. The result of a match between the client token and the reference source token may prompt the client to assign a new record ID to reflect the existence of a match.

[0052] Figure 5Exemplary data sources and tokens are shown. Environment 500 shows one or more data sources 502, 504, and 506. One or more data sources 502, 504, and 506 may all be owned by the same client or different clients. In one example, one or more data sources 502, 504, and 506 may be owned by the same client (e.g., a bank) and associated with different programs from the client (e.g., data source 502 may be residential home loan data, while data source 504 may be personal checking account data). Each of these data sources has a token, which is a tokenized (i.e., hashed / encrypted) data attribute, such as name, DOB, address, SSN, gender, email, phone, etc., as previously described. Each token (e.g., T1, T2, T3, ..., Tn, etc.) may characterize multiple data attributes, a single data attribute, or a portion of a data attribute, as previously explained. These tokens are then compared to each other from record to record to determine if there is a match between the tokens of the source records, which may produce a token set (anywhere the same token is found between source records).

[0053] Figure 6 Examples of matching token pairs and matching set token pairs, establishing token set rules, and matching token sets are shown. Environment 600 can include matching comparison 602, in which tokens at the record level are compared with each other. For example, record 1 token 1 is compared with record 2 token 1. According to matching comparison 602, matching pairs 604 can be established. For example, a matching pair can be generated due to record 1 and record 2 if these records all have token 1. These matching pairs are then compared with set rules 606. Set rules can be established to ensure high matching accuracy. For example, a set rule can be T1+T2+T4, which means that if two records each having token 1, token 2, and token 4 match each other, a token set can be generated. Another rule can include T3+T5+T7, which means that if a token record each has tokens 3, 5, and 7, a token set can be generated.

[0054] Once a set of token records is determined to satisfy the set rules 606 (i.e., simple weighting), the entity may be said to be "resolved" or the records may be determined to match (either duplicates in the case of records from the same source or overlaps in the case of records from different sources). In other instances, instead of comparing token by token and then checking the set rules to determine a match, tokens may be assembled according to the set rules for each record at 608 and re-tokenized into a single set token SeT (hash accumulation). Hash accumulation reduces token pair comparisons, for example, for SeT1, from three comparisons according to the set rules (T1+T2+T4) in the first case of simple weighting to a single comparison. Once a set is generated at 608, the set may be applied to future tokens ingested by the system. Thus, if a record containing token 1, token 2, and token 4 is received, the token record may be matched to the already established token set S1 (T1+T2+T4). In some cases, hash accumulation may also be applied to duplicate detection in a manner similar to the above explanation of overlap detection.

[0055] To provide a discrete example, a set rule can be defined like this:

[0056] S1: T1+T2+T4

[0057] The tokens in the data set can be T1-T4, and T1-T4 can each be a tokenized version of the underlying token attributes. T1-T4 can represent the following data attributes (but note that the system cannot see the underlying data attributes because these data attributes are tokenized):

[0058] T1: Name + DOB + Email

[0059] T2: Name

[0060] T3: SSN+DOB

[0061] T4: License number + CC number

[0062] When tokens are received by a system (e.g., a third-party integration platform), the tokens can be matched against each other at the record level to create token sets:

[0063] Source 1 (S1)

[0064] R1: T1 T2 T3

[0065] R2: T1 T2 T3 T4 T5

[0066] R3: T1 T2 T4

[0067] Source 2 (S2)

[0068] R1: T1 T3

[0069] R2: T1 T2 T4 T5

[0070] R3: T1 T2 T4 T5

[0071] Source 3 (S3)

[0072] R1: T1 T3

[0073] R2: T2 T3 T5

[0074] R3: T1 T2 T4

[0075] In the above example, records satisfying the set rule (T1+T2+T4) will include the following sets: Duplicates - S1R2, S1R3, Overlaps - S1R2, S1R3, S2R2, S2R3, S3R3. Each of these tokens will be associated with a single set rule: Rule 1 (T1+T2+T4).

[0076] In other aspects, tokens may be accumulated into set tokens by applying predetermined rules based on a desired source or reference source match:

[0077] Rule 1 = T1 + T2 + T4

[0078] Set token according to rule 1 (SeT1) = T (T1+T2+T4)

[0079] Source 1 (S1)

[0080] SeT1

[0081] SeT2

[0082] SeTX

[0083] Source 2 (S2)

[0084] SeT2

[0085] SeTY

[0086] ZGar

[0087] Source 3 (S3)

[0088] SeT1

[0089] SeT2

[0090] XT

[0091] In the above example, source records that meet certain criteria of a set rule can be automatically accumulated into token sets (tokens within tokens) rather than being compared individually with other tokens at the record level, thereby reducing the time required to create possible matching pair combinations for the number of available tokens for each record.

[0092] The hash accumulation process explained above can be used to find overlaps between different source records and between source records and reference records, and can also be used to find duplications between records of the same source.

[0093] Figure 7 An exemplary architecture for high performance data matching is shown. The architecture 700 includes a match preprocessor 702, a match processor 704, a database loader 706, a database 708, and an entity merge processor 710. The match preprocessor 702 is configured to generate a partition file consisting of hashed (e.g., only hashed tokens), masked (e.g., hashed and static encrypted tokens), and / or encrypted (e.g., only encrypted tokens) tokens. These tokens are then transmitted to the match processor 704, where the partition token file is used to generate intermediate files during the matching process. These intermediate files are suitable for consumption by a database loading process, where the file represents the matching results of a single source containing entities, sets, and records. For example, a client can upload tokenized records from a single data source, and the match processor can determine overlapping tokenized records and integrate these tokenized records (e.g., tokenized composites).

[0094] The matched tokens may then be passed to a database loader 706, where the database loader consumes the database prepared matched source files. The database loader may be a batch database loader, a JD BC database loader, and other similar types of database loaders. The database loader then passes the database prepared files to a database 708, such as a PostgreSQL database, where the tokens are saved across multiple data sources.

[0095] The entity merge processor 710 may call the database 708 and start matching the tokenized entities to each other to create a token set. The entity merge processor 710 may be configured with a recursive database function to match aggregated tokens at the record level to create a record set.

[0096] Figure 8An example of a suitable operating environment in which one or more embodiments of the present embodiment can be implemented is shown. This is only an example of a suitable operating environment and is not intended to limit the scope of use or functionality. Other well-known computing systems, environments, and / or configurations that may be suitable for use include, but are not limited to, personal computers, server computers, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, programmable consumer electronics (e.g., smartphones), network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like.

[0097] In its most basic configuration, the operating environment 800 typically includes at least one processing unit 802 and memory 804. Depending on the exact configuration and type of computing device, the memory 804 (which stores information about detected devices, related information, personal gateway settings, and instructions for executing the methods disclosed herein, etc.) can be volatile (e.g., RAM), non-volatile (e.g., ROM, flash memory, etc.), or some combination of the two. Figure 8 In the embodiment of the present invention, the environment 800 may be provided with a plurality of input devices 814, such as a keyboard, a mouse, a pen, a voice input device, etc., and / or one or more output devices 816, such as a display, a speaker, a printer, etc. The environment 800 may also include one or more communication connections 812, such as LAN, WAN, point-to-point, etc.

[0098] The operating environment 800 typically includes at least some form of computer-readable media. Computer-readable media can be any available media that can be accessed by the processing unit 802 or other devices that include the operating environment. By way of example, and not limitation, computer-readable media can include computer storage media and communication media. Computer storage media include volatile and non-volatile removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disk (DVD), or other optical storage, cassettes, magnetic tape, disk storage, or other magnetic storage devices, or any other tangible media that can be used to store the desired information. Computer storage media do not include communication media.

[0099] Communication media embodies non-transitory computer-readable instructions, data structures, program modules, or other data. Computer-readable instructions may be transmitted in a modulated data signal such as a carrier wave or other transport mechanism, and include any information delivery media. The term "modulated data signal" refers to a signal that has one or more of its characteristics set or changed in a manner that encodes the information in the signal. By way of example, and not limitation, communication media include wired media, such as a wired network or a direct wired connection, and wireless media, such as acoustic, RF, infrared, and other wireless media. Combinations of any of the above should also be included within the scope of computer-readable media.

[0100] Operating environment 800 can be a single computer that operates in a networked environment using a logical connection to one or more remote computers. The remote computer can be a personal computer, a server, a router, a network PC, a peer device, or other public network nodes, and typically includes many or all of the above elements and other elements that are not mentioned in this way. The logical connection can include any method supported by an available communication medium. Such a networked environment is often located in an office, an enterprise-wide computer network, an intranet, and the Internet.

[0101] For example, the above text describes various aspects of the present disclosure with reference to the block diagrams and / or operational diagrams of the methods, systems, and computer program products according to various aspects of the present disclosure. The functions / actions marked in the blocks may not occur in the order shown in any flow chart. For example, two blocks shown in succession may in fact be substantially executed concurrently, or the blocks may sometimes be executed in reverse order, depending on the functions / actions involved.

[0102] The description and illustration of one or more aspects provided in this application are not intended to limit or restrict the scope of the disclosure claimed for protection in any way. The multiple aspects, examples, and details provided in this application are considered to be sufficient to convey the possession thereof, and enable others to make and use the best mode of the disclosure claimed for protection. The disclosure claimed for protection should not be interpreted as being limited to any aspect, example, or detail provided in this application. Whether shown and described in combination or individually, a variety of features (both structures and methods) are intended to be selectively included or omitted to produce an embodiment with a specific feature set. After providing the description and illustration of this application, those skilled in the art can envision changes, modifications, and alternative aspects that fall within the spirit of the broader aspects of the general inventive concept embodied in this application, without departing from the broader scope of the disclosure claimed for protection.

[0103] In summary, it should be understood that specific embodiments of the present invention have been described herein for illustrative purposes, but that various modifications may be made without departing from the scope of the present invention. Therefore, the present invention is not to be limited except as set forth in the appended claims.

Claims

1. A system for data matching, comprising: a memory configured to store non-transitory computer-readable instructions; as well as a processor communicatively coupled to the memory, wherein the processor, when executing the non-transitory computer-readable instructions, is configured to: receiving at least one token record from a first source; receiving at least one token record from a second source; comparing the at least one token record from the first source to the at least one token record from the second source; determining, based on the comparison of the at least one token record from the first source and the at least one token record from the second source, at least one common token in the at least one token record from the first source and in the at least one token record from the second source, wherein the at least one common token is associated with at least one set rule; matching the at least one token record from the first source with the at least one token record from the second source based on the at least one set rule; and A token set is generated by merging the at least one token record from the first source with the at least one token record from the second source using at least one hash accumulation function, wherein the token set includes the at least one token record from the first source and the at least one token record from the second source.

2. The system of claim 1, wherein the processor is further configured to: comparing the set of tokens to at least one universal reference token repository; and Based on determining that the set of tokens is not present in the at least one universal token repository, the set of tokens is stored in the universal reference token repository.

3. The system according to claim 1, wherein: The at least one token record from the first source is associated with at least two tokens.

4. The system according to claim 1, wherein: The at least one token record from the second source is associated with at least two tokens.

5. The system according to claim 2, wherein: The at least one universal reference token repository includes at least one of: client tokens and reference source tokens.

6. The system according to claim 1, wherein: The first source and the second source are owned by a common owner.

7. The system according to claim 1, wherein: The set rules include at least two token requirements, wherein one of the at least two token requirements is the presence of the at least one common token.

8. The system according to claim 1, wherein: The token set includes a plurality of token records, wherein each token record of the plurality of token records includes the at least one common token.

9. The system according to claim 1, wherein: The first source and the second source are at least one of: a customer source and a reference source.

10. The system according to claim 1, wherein: The processor is further configured to: At least one token record is received from a third source, wherein the at least one token record from the third source includes the at least one public token.

11. The system according to claim 10, wherein: The processor is further configured to: matching the at least one token record from the third source to the token set, wherein the token set is based on the set rule, wherein the set rule requires the at least one common token; and The at least one token record from the third source is merged with the token set.

12. A method for data matching, comprising: receiving at least one token record from a first source; receiving at least one token record from a second source; receiving at least one token set rule, wherein the at least one token set rule is based on the presence of at least one common token; comparing the at least one token record from the first source to the at least one token rule; comparing the at least one token record from the second source to the at least one token rule; determining, by simple weighting, that the at least one token record from the first source and the at least one token record from the second source satisfy the at least one token rule; merging the at least one token record from the first source with the at least one token record from the second source using at least one hash accumulation function; and A token set is generated, wherein the token set includes the merged at least one token record from the first source and the at least one token record from the second source.

13. The method according to claim 12, wherein: The at least one token record from the first source and the at least one token record from the second source are encrypted using the same encryption algorithm.

14. The method according to claim 12, further comprising: comparing the set of tokens to at least one universal reference token repository; as well as Based on determining that the set of tokens is not present in the at least one universal token repository, the set of tokens is stored in the universal reference token repository.

15. The method according to claim 14, wherein: The universal reference token repository includes a plurality of token records from at least one of: a customer source and a reference source.

16. The method according to claim 12, further comprising: comparing at least one token record from a third source to the token set; as well as The at least one token record from the third source is re-tokenized into the token set by at least one hash accumulation function.

17. The method according to claim 16, further comprising: At least one duplicate token record from the same source is determined based on the at least one hash accumulation function.

18. The method according to claim 16, further comprising: At least one overlapping token record in the third source is determined based on the at least one hash accumulation function.

19. A computer readable medium storing non-transitory computer executable instructions which, when executed, cause a computing system to perform a method for data matching, the method comprising: receiving at least one token record from a first source, wherein the at least one token record from the first source includes at least one public token; receiving at least one token record from a second source, wherein the at least one token record from the second source includes the at least one public token; receiving at least one token set rule, wherein the at least one token set rule is based on the presence of the at least one common token; comparing the at least one token record from the first source to the at least one token rule; comparing the at least one token record from the second source to the at least one token rule; determining, based on the presence of the at least one common token, that the at least one token record from the first source and the at least one token record from the second source satisfy the at least one token rule; merging the at least one token record from the first source with the at least one token record from the second source using at least one hash accumulation function; and A token set is generated, wherein the token set includes the merged at least one token record from the first source and the at least one token record from the second source.

Images