Data warehouse native personally identifiable information (PII) resolution

Abstract

By utilizing user-defined functions in the structured query language SQL, a system for performing entity resolution, specifically ranked personally identifiable information (PII) matching, can function directly within the data warehouse's execution environment, thereby eliminating the need to move data to external systems. This system uses SQL and user-defined function (UDF) implementations to utilize an inverted index system that can operate on the distributed processing capabilities of the data warehouse.

Description

Technical Field

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 462,144, filed Apr. 26, 2023. The above application is hereby incorporated by reference in its entirety.

[0002] Description of Research or Development Funded by the Federal Government Not applicable.

Background Art

[0003] Personally identifiable information (PII) is a type of data that can be used to identify a person's identifying information, including name, address, phone number, etc. PII entity resolution processes data and merges matching records. Matching PII records from a source dataset to a target dataset is an important part of the marketing technology industry as it enables marketers to discover and target addressable individuals with high accuracy. There are many systems that perform PII entity resolution, which compare and match sets of data stored across multiple channels.

[0004] An inverted index is a type of data structure that stores and organizes information to enable efficient search. Inverted indexes are frequently used in information retrieval systems. Lucene and ElasticSearch are common systems that provide inverted index functionality. However, these systems cannot be run natively within modern data warehouses such as BigQuery, Snowflake, and SingleStore.

[0005] The references described in this background art section are not admitted to be prior art with respect to the present invention.

Summary of the Invention

[0006] This invention relates to a system for performing entity resolution, specifically ranked PII matching, within the execution environment of a data warehouse. While many systems exist for performing PII entity resolution, this invention enables ranked matching of PII records directly within a data warehouse or distributed database (e.g., BigQuery, Snowflake, SingleStore, Redshift), thereby eliminating the need to move data to an external system. This invention utilizes an inverted index system that can operate with the distributed processing capabilities of a data warehouse using a structured query language (SQL) and user-defined function (UDF) implementation.

[0007] Implementing PII resolution within a data warehouse offers the following advantages: Firstly, highly sensitive PII data no longer needs to leave the warehouse compute environment. This allows users to fully leverage the data lineage and governance capabilities of their warehouse without incurring the additional risks associated with implementing resolution using external systems. Secondly, data warehouses often provide easily scalable or "serverless" compute environments, enabling users to optimize their cloud compute usage with minimal effort.

[0008] These and other features, purposes, and advantages of the present invention will be better understood in consideration of the following detailed description of preferred embodiments and the appended claims, together with the drawings, as described below. [Brief explanation of the drawing]

[0009] [Figure 1] This is a flowchart providing an overview of a prospect scoring method according to one embodiment of the present invention. [Figure 2] This is a flowchart for a manufacturing system method according to one embodiment of the present invention. [Figure 3] This figure shows a system architecture according to one embodiment of the present invention. [Figure 4] This is a flowchart for a method for constructing a prospect token index according to one embodiment of the present invention. [Figure 5] This is a flowchart for a method of creating and matching customer records according to one embodiment of the present invention. [Modes for carrying out the invention]

[0010] Before the present invention is described in further detail, it should be understood that the scope of the present invention is limited only by the claims, and that the invention is not limited to the specific embodiments described, nor is it limited to the specific embodiments, and that the terms used in describing those specific embodiments are merely for the purpose of describing those specific embodiments and not to limit them.

[0011] The objective of this invention is to match two tables containing rows of PII records with each other. In one embodiment, Table A is called the reference dataset and Table B is called the input dataset. Since PII records are generally not standardized and may contain errors, a simple inner join between Table A and Table B would result in a lower match rate than expected. Instead of matching the entire record, this invention generates many combinations of tokens from each record. Tokens are generated by splitting the PII data elements by whitespace. These token combinations can then be used as join keys in a data warehouse to produce a higher match rate.

[0012] In one example of one embodiment, JOHN L SMITH II 123 MAIN ST is a record from Table A. This record contains seven tokens: four tokens for the name and three tokens for the street address. The following 15 combinations of name tokens can be generated. JOHN L SMITH II JOHN L SMITH II JOHN L II JOHN SMITH II L SMITH JOHN L JOHN SMITH II JOHN L SMITH II L II SMITH JOHN L SMITH II The following seven combinations of street address tokens can be generated. 123 Main Street 123 MAIN 123 ST MAIN ST one two three MAIN ST At this stage, the order of the tokens is not important, and therefore the tokens are sorted lexicographically. Each combination of a name token and a street address token is further combined to produce a total of 105 combinations.

[0013] In the same example, consider the following records from Table B. JOHN SMITH 123 MAIN RD Using the same technique for token generation, the record contains a set of 21 tokens. When performing an inner join of two sets of tokens, the following match will be generated: JOHN SMITH MAIN SMITH 123 MAIN JOHN SMITH 123 MAIN JOHN SMITH 123 JOHN 123 MAIN JOHN 123 JOHN MAIN SMITH MAIN SMITH 123

[0014] There is a potential or prospect match between the records of Table A and the records of Table B. Since this is only considered a potential match, the scoring process 10 will use the match rule engine 12, as shown in FIG. 1, to compare the original Table A records with the Table B records to make a final determination as to whether this is a valid match. This process for generating prospect matches will further be optimized by a very large number of record and token combinations that may exist in Table A and Table B.

[0015] The first optimization is to rank the quality of prospect matches based on the degree of their readiness for use. This can be achieved by assigning a score to each token that takes into account the number of tokens and the frequency of the tokens that make up the prospect token. System 10 does not need to return all matches of token combinations between two records. Rather, System 10 only needs to consider the highest scoring token combination match for a given prospect. Further, in certain embodiments, System 10 will only consider the top 15 highest scoring prospects. The top 15 prospect matches must further be filtered before determining whether a suitable match has been found. The match rule engine 12 implements this matching function, as shown in FIG. 1.

[0016] As shown in Figure 1, the match rule engine 12 is natively located within the data warehouse, and therefore the data is not moved outside the warehouse's execution and storage environment. System 10 will generate match engine code that can be executed within the user-defined function (UDF) execution environment of the data warehouse in order to implement the match rule engine 12.

[0017] Figure 2 shows the source code, configuration, and data flow required to transpile match rules into a Java® script UDF executed in BigQuery. The match rule engine 12 is a programmatic framework for defining and executing match rules. Match rules are designed to return a numerical score representing the probability that two records are a match. Match rules are defined in XML format 14 and executed by the match rule engine 12. The match rule engine is designed to be extensible and can be customized to meet specific user needs or to adapt to changes in data.

[0018] Match rules are composable and can be composed in any combination. Therefore, the engine is a general-purpose computing framework. The method by which the engine combines match rules is translated into Python code in the rule transpiler 13, and the rule transpiler 13 can automatically generate Java(registered trademark) script code as match translation rules 15 that can execute the logic defined in the rule XML 14.

[0019] BigQuery boilerplate code 17 is a top-level interface between BigQuery UDF definitions and Java® script code executed within the UDFs. It is integrated into the final match UDF definition 20 as part of the interface specification.

[0020] The match rule engine 12 includes definitions of how primitives should be concatenated when calculating a numerical probability score for matching two records. The rule XML configuration 14 is read by the PY rule transpiler 13, and thus Java® script code (logical flow, i.e., if-else statements, primitive concatenation, and interface definitions) can be generated as match translation rules 15.

[0021] Filter Primitive 16 is a library of code primitives whose use can be configured in XML when comparing a prospect to an input record. Functionality types include, but are not limited to, append, pluralize, split, edit distance, IsNickName, etc. The Java® implementation of Primitive Function 16 is used by Rule XML 14. All primitive functions share the same interface and therefore can be composed in any combination.

[0022] The lookup table 18 is a manually curated dataset used as reference data when several primitives compare two records, and does not necessarily include a nickname table, a gender lookup table, and a street suffix table. The match rule engine 12 takes the rule XML 14, filter primitives 16, and lookup table 18 as input and creates a match UDF 20 and SQL interface that fits the target data warehouse. For example, a table in lookup table 18 might contain a list of first names and a corresponding set of nicknames. Other examples include a list of first names and a list of highly likely genders for an individual with that name, or a list of postal address street suffixes and common abbreviations for those street suffixes.

[0023] As shown in Figure 3, the system architecture comprises the following tables and steps: a reference graph 22, a prospect token index 24, a customer input record / table 26, a customer input token 28, a matched token 30, a match engine UDF 20, and a matched record 34. The system architecture supports the ability to push down the aggregation and filtering of query results to leaf processing nodes before subsequent aggregation and filtering occurs on the aggregator nodes. This is a key feature for scaling the invention, as it allows each leaf processing node to determine the highest-scoring token combination for a given prospect and filter all records except the top 15 highest-scoring prospects local to the leaf node before sending the results to the aggregator node. Generalizing this information retrieval method to any short-length document store (10 words or less per document) results in a massively parallel inverted index of short documents natively implemented within the data warehouse.

[0024] Reference graph 22, in this embodiment, contains a short document store containing PII entity records for name, postal address, email address, and telephone number. Each entity record is stored as a set of PII tokens. Each record in this table is converted into its set of unique token combinations using SQL. In one example, reference graph 22 may contain approximately 10 billion name and postal address records.

[0025] The Prospect Token Index 24 is a table where each row contains a unique combination of tokens for one of the PII short document records. This table serves as the primary index for searching for prospect PII matches. In one example, the Prospect Token Index 24 may contain approximately 600 billion token combinations.

[0026] Customer input record 26 contains a set of PII records that a typical user will use as their consumer knowledge base. Each PII record is converted into a set of unique token combinations using SQL. These token combinations are generated in the same format as reference graph 22. In one example, customer input record 26 may contain up to 50 million name and postal address records.

[0027] The customer input token table 28 is a table where each row contains a unique combination of tokens for one of the PII customer records. This table will be joined to the prospect token index 24 to find prospect PII matches. In one example, the customer input token table 28 may contain up to 3 billion token combinations.

[0028] Matched Tokens 30 is a table that is the result set from an inner join between Prospect Token Index 24 and Customer Input Token 28. This result set contains up to 15 prospect matches for each customer record. A prospect match is a PII record that matches all or part of a PII token. Prospect matches are ordered by a numerical match score, which represents an approximation of how strictly those records match. In one example, Matched Tokens 30 may contain up to 750 million prospect matches.

[0029] As previously described, the match engine UDF20 compares each of the 15 prospects to the original customer record and returns a score and final judgment on whether the record meets all the matching criteria defined in Rule XML14. Therefore, the match engine UDF20 uses the match rule engine 12 to filter prospect matches.

[0030] The table of matched records 34 contains the original customer PII records that were matched in the reference graph 22. The information in this table is limited to identifiers pointing to the records and metadata regarding the quality and strength of the matches performed.

[0031] As shown in Figure 4, the method for constructing the prospect token index includes the following tables and steps, namely, the reference graph 22, the normalized reference graph 36, the normalized tokens 38, the reverse document frequency 40, the document norm 42, and the weighted tokens 44, resulting in the prospect token index full 25 and the (final) prospect token index 24. The reference graph 22 contains all third-party matchable entities. In one embodiment, using a large consumer database, this table may contain tens of billions of rows. The reference graph 22 includes the following data fields: ENTITYID CONSUMERLINK - (A unique identifier for an individual) FIRSTNAME MIDDLENAME LASTNAME GENERATIONALSUFFIX STREETNUMBER STREETPREDIRECTIONAL STREET STREETSUFFIX STREETPOSTDIRECTIONAL UNITDESIGNATOR SECONDARYNUMBER ZIPCODE CITY STATE Next, the table passes through a normalization UDF, which performs basic data transformations, such as standardizing the zip code to 5 digits and removing extra whitespace in other PII fields. As a result, the normalized reference graph 36 contains the same data fields as reference graph 22.

[0032] The token is obtained by splitting each PII field by whitespace. The normalized token 38 contains the following data fields: entityId consumerLink Field Type token match_partition The field "fieldType" indicates a name or street address and is used to calculate normalized weights representing the matching value to a given token. The field "match_partition" points to the partition key used for matching purposes, which in one embodiment is a five-digit zip code. This is different from the more common partition_key used to partition data storage within a data warehouse. Two nominal values ​​are calculated for each token by grouping rows by (fieldType, token, match_partition). Entities from the normalized reference graph 36 are burst into one row for each PII token. Tokens are obtained by splitting each PII field by whitespace. fieldType can be either a name or a street address and will later be used to calculate normalized weights representing the matching value to a given token.

[0033] As a result of the process described above, the reverse document frequency 40 will, in that case, include the following data fields: entityId consumerLink Field Type token match_partition numDocumentsWithToken idfT By grouping rows by (fieldType, token, match_partition), two nominal values ​​are calculated for each token. The field "numDocumentsWithToken" is a simple frequency metric. The field "idfT" refers to the inverse document frequency of the token, calculated as log(totaldocuments / (numDocumentsWithToken+1)).

[0034] Document norm 42 includes the following data fields: entityId consumerLink Field Type token match_partition numDocumentsWithToken idfT documentNorm The field "documentNorm" is the final calculated value, which assigns weights to the importance of matching to a given field (fieldType, token, match_partition). The weights are inversely proportional to the frequency of the tokens; therefore, more common values ​​have smaller weights than rarer tokens. The tokens are then aggregated with their respective entities by grouping them by entityId. The weighted tokens 44 contain the following data fields: entityId match_partition consumerLink nameTokens StreetAddressTokens nameTokens are generated when PII name information is split by whitespace. For example, the string "JOHN L SMITH" becomes a set of tokens "JOHN", "L", and "SMITH". streetAddressTokens are similarly generated when street address information is split by whitespace to produce a set of streetAddress tokens.

[0035] For each entity and its set of nameTokens and streetAddress tokens, all combinations of selecting k tokens from n (n choose k) are generated and scored. This allows for the implementation of a distributed inverted index directly within the data warehouse, which is uniquely achievable in this domain due to the relatively small number of tokens (usually fewer than 10) that an entity representation may contain. The scoring function uses the documentNorm of each token to calculate a score representing how likely an entity is to be matched. This score allows for the ranking of all potential matches and efficiently prunes lower-ranked entities. The Prospect Token Index Full 25 includes the following data fields: match_partition consumerLink entityId download token partition_key The partition key is calculated in such a way that the amount of data queried when performing a match can be significantly reduced, which leads to increased performance and reduced cost for calculating the results. In this embodiment, the partition_key is calculated from the postal code of the postal address. In some cases, the partition_key can be the postal code itself, and in other cases, the modulo division operator is used to reduce the total number of partitions to a value that fits the data warehouse.

[0036] Ultimately, within the match partition, identical token combinations from different prospects that exceed rank 15, ordered by score, are removed. For example, 123 main token combinations may exist among hundreds of prospects, but only the top 15 of those token combinations need to be considered, because the others never exceed the cutoff. This is an optimization for pruning tokens that never match within the match partition. In alternative embodiments of the present invention, different cutoff numbers may be used. The number of common token combinations considered when discovering prospects is significantly reduced in this way, which leads to a reduction in the amount of data scanned, shuffled, and aggregated. This results in faster and more efficient matching queries on the data warehouse. The result is a prospect token index 24, which has the same type of value as the prospect token index full 25.

[0037] As shown in Figure 5, the prospect and match process includes the following tables and steps: customer input record / table 26, normalized customer table 46, customer prospect token 48, customer prospect result 50, customer prospect PII 52, and customer match result 54. The customer input table 26 contains the set of customer records to be matched with the reference graph 22. The main objective of the matching process is to determine the best consumerLink for each row in this table. The customer input table 26 includes the following data fields: uuid name Street Address city state zipcode A UUID is a universally unique identifier commonly used in computer systems, and it can be implemented as a 128-bit value. The PII data from the previous table is normalized and tokenized by separating each field with whitespace. The normalized customer table 46 contains the following data fields: uuid nameTokens StreetAddressTokens match_partition

[0038] For each record and its set of nameTokens and streetAddress tokens, all combinations of selecting n to k tokens are generated. These tokens are used as internal join keys to prospect token index 24. The customer prospect token 48 contains the following data fields: uuid token match_partition The customer prospect token table 44 is internally joined with the prospect token index 24. The internal join is grouped by (uuid, entityId) to determine the highest match score for entityId. Then, in one embodiment, these results are grouped by uuid to determine the top 15 scored entityIds for uuid. This method allows most distributed data warehouses to perform local aggregation and filtering on leaf nodes before shuffling the data on aggregator nodes. The result is an extremely efficient pruning of invalid matches that can scale horizontally to match thousands of nodes. The customer prospect results 50 include the following data fields: uuid prospect.entityId prospect.consumerLink prospect.score

[0039] In Customer Prospect PII52, once the top 15 prospects are identified for each input record, in one embodiment, each UUID is joined to its original PII in the customer input record 26, and each entityId is joined to its original PII from the match rule engine 12. Customer Prospect PII52 includes the following data fields: customerRecord.uuid customerRecord.name customerRecord.streetAddress prospect.score prospect.ENTITYID prospect.CONSUMERLINK prospect.FIRSTNAME prospect.MIDDLENAME prospect.LASTNAME prospect.GENERATIONALSUFFIX prospect.STREETNUMBER prospect.STREETPREDIRECTIONAL prospect.STREET prospect.STREETSUFFIX prospect.STREETPOSTDIRECTIONAL prospect.UNITDESIGNATOR prospect.SECONDARYNUMBER prospect.ZIPCODE prospect.CITY prospect.STATE

[0040] The customer match result 54 is the result of passing the input record and the top 15 prospects to a UDF containing the match rule engine 12. The UDF is automatically generated and produced by creating code that can be executed within the data warehouse UDF execution environment, including, but not limited to, Java Script, Python, and web assembly. The automatically generated code is a port of the match rule engine 12 logic.

[0041] In one embodiment of the present invention, the logical UDF components of the present invention each comprise the following: 1. UDF for normalizing PII data, 2. UDF for calculating the total number of documents per partition. 3. UDF for calculating the inverse document frequency of tokens, 4. UDF for calculating the total tokens for each field. 5. UDF for calculating normalized weights for each token in a document. 6. UDF for generating scored prospect token permutations, 7. UDFs for preparing customer data for prospect token matching. 8. UDF for discovering prospect-matched AbiliTec entities for input entity tokens, and 9. A UDF for performing matching logic between input records and prospect records.

[0042] In one embodiment of the present invention, the minimum requirements for operation in the creation environment include: the warehouse must have the capacity to store at least 100 TB of data; the warehouse must have the capacity to store at least 1 trillion rows of data; the warehouse must have the capacity to store at least 100 columns of data; the warehouse must have the capacity to execute non-SQL UDF code such as, but not limited to, Java Script, Java, Python, and WASM; and the warehouse must have the capacity to perform standard SQL joins and combinations of SQL joins, namely LEFT, RIGHT, OUTER, and INNER.

[0043] The methods described herein can be implemented in various embodiments by any combination of hardware and software. For example, in one embodiment, the method may be implemented by a computer system or a collection of computer systems, each including one or more hardware processors that execute program instructions stored in a computer-readable physical storage medium coupled to a hardware processor. The program instructions may implement the functionality described herein (for example, the functionality of various hardware servers and other components that implement the network-based cloud and non-cloud computing resources described herein). The various methods shown in the figures and described herein represent exemplary implementations. The order of any method may be changed, and various elements may be added, modified, or omitted.

[0044] A computer system can be any of various types of hardware devices, including, but is not limited to, commodity servers, personal computer systems, desktop computers, laptop or notebook computers, mainframe computer systems, handheld computers, workstations, network computers, consumer devices, application servers, physical storage devices, telephones, mobile phones, or generally any type of computing node, compute node, compute device, and / or hardware computing device.

[0045] Any of the distributed system embodiments described herein, or any of their components, may be implemented as one or more network-based services in a cloud computing environment. For example, read-write nodes and / or read-only nodes in the database tier of a hardware database system may present database services and / or other types of physical data storage services employing the distributed storage systems described herein to clients as network-based services. In some embodiments, network-based services may be implemented by software and / or hardware systems designed to support interoperable machine-to-machine interaction over a network. Web services may have interfaces written in a machine-readable format. Other systems may interact with network-based services in a manner defined by the description of the network-based service's interface. For example, a network-based service may define various actions that other systems may invoke and define specific application programming interfaces (APIs) that other systems are expected to follow when requesting these actions.

[0046] In various embodiments, network-based services may be requested or invoked through the use of a message containing parameters and / or data associated with the network-based service request. Such messages may be formatted according to a specific markup language, such as XML, and / or encapsulated using a protocol. To fulfill a network-based service request, a network-based service client may assemble a message containing the request and transmit that message to an addressable endpoint corresponding to the web service (e.g., a Uniform Resource Locator (URL)) using an Internet-based application layer transport protocol, such as the Hypertext Transfer Protocol (HTTP).

[0047] Unless otherwise specified, all technical and scientific terms used herein have the same meaning as those generally understood by those skilled in the art to the extent of this invention. Similar or equivalent methods and materials as those described herein may also be used in the implementation or testing of this invention, but only a limited number of exemplary methods and materials are described herein. It will be apparent to those skilled in the art that many more modifications are possible without departing from the inventive concept herein.

[0048] All terms used herein should be interpreted in the broadest possible form that is appropriate to the context. In particular, the terms “comprises” and “comprising” should be interpreted in a non-exclusive manner as referring to an element, component, or step, indicating that the referenced element, component, or step may be combined with other elements, components, or steps that exist, are used, or are not explicitly mentioned. When grouping is used herein, all individual members of the group, as well as all possible combinations and subcombinations of the group, are included individually. When a scope is specified herein, the scope includes all subscopes within the scope, as well as all individual points within the scope. When terms such as “about” and “approximately” are used herein, the terms should include quantities, measurements, etc. that do not deviate significantly from explicitly stated quantities, measurements, etc., so as not to impair the specified purpose of the apparatus or process. All references cited herein are incorporated herein by reference to the extent that they do not conflict with this disclosure.

[0049] The present invention has been described in relation to several preferred and alternative embodiments, which are intended to be illustrative and not to be limited to the full scope of the invention as described in the appended claims.

Claims

1. A method for matching personally identifiable information (PII) records within a data warehouse execution environment, A step of receiving a source dataset in the data warehouse execution environment, wherein the source dataset includes personally identifiable information (PII), The steps include generating an index of prospect token combinations from the aforementioned source dataset, The steps include generating prospect token matches from the index of token combinations using a match engine natively located within the data warehouse execution environment, The steps include filtering the aforementioned token matches to create a set of filtered token matches, The steps include creating a matched customer record from the filtered token match and A method for matching personally identifiable information records within a data warehouse execution environment, including the following.

2. The step of generating an index of prospect token combinations is: Steps to merge third-party matchable entities, The steps include passing the third-party matchable entity through a set of normalization user-defined functions that perform data transformation, The steps include: breaking down each token into a single entity, The steps include calculating the inverse document frequency of the token, The steps include: calculating the total tokens for each field, The steps include: calculating the normalized weight for each token in the document, The steps include generating a scored token permutation and The method according to claim 1, further comprising:

3. The method according to claim 2, further comprising the step of preparing customer data for matching tokens.

4. The method according to claim 3, further comprising the step of filtering prospect match entities with respect to input entity tokens.

5. The method according to claim 4, further comprising the step of performing matching between input records and prospect records within the data warehouse execution environment.

6. The method according to claim 5, further comprising the step of applying a cutoff value to the scored token permutations and excluding scored token permutations that fall below the cutoff value.

7. A cloud computing environment comprising one or more computer processors and memory space storing instructions, wherein when an instruction is executed by the one or more computer processors, the one or more computer processors: Receiving source datasets containing personally identifiable information (PII), To generate an index of prospect token combinations from the aforementioned source dataset, Generating prospect token matches from the aforementioned index of token combinations, The aforementioned token matches are filtered to create a set of filtered token matches, From the filtered token matches, create a matched customer record. A cloud computing environment that enables this process.

8. When executed by the one or more computer processors, the instruction for generating an index of prospect token combinations, when executed by the one or more computer processors, to the processor, Merging third-party matchable entities, The third-party matchable entity is passed through a set of normalization user-defined functions that perform data transformation, The entity is broken down into one line for each token, Calculating the inverse document frequency of tokens, Calculate the total tokens for each field, Calculating normalized weights for each token in the document, To generate scored token permutations and The cloud computing environment according to claim 7, further comprising commands to perform the following:

9. The cloud computing environment according to claim 8, wherein when the instruction is executed by the one or more computer processors, the one or more computer processors are further instructed to prepare customer data for matching tokens.

10. The cloud computing environment according to claim 9, wherein when the instruction is executed by the one or more computer processors, the one or more computer processors are further instructed to filter prospect match entities with respect to input entity tokens.

11. The cloud computing environment according to claim 10, wherein when the instruction is executed by the one or more computer processors, the one or more computer processors are further instructed to perform matching between input records and prospect records.

12. The cloud computing environment according to claim 11, wherein when the instruction is executed by the one or more processors, the one or more computer processors further cause the computer processors to apply a cutoff value to the scored token permutations and to remove scored token permutations that fall below the cutoff value.

13. A user-defined function (UDF) for total documents, configured to calculate the total number of documents per partition in a set of data containing personally identifiable information (PII), A reverse document frequency UDF configured to calculate the reverse document frequency of tokens in the aforementioned document, A total token UDF configured to calculate the total number of tokens for each field in the aforementioned document, A normalized weight UDF configured to calculate a normalized weight for each token within each of the aforementioned documents, With respect to the aforementioned document, a scored prospect token permutation UDF configured to generate scored prospect token permutations, A customer data preparation UDF configured to prepare customer data for matching with prospect tokens in the aforementioned document, A prospect match entity discovery UDF configured to discover prospect match entities for the aforementioned prospect token, A matching logic UDF configured to perform matching logic between input records and prospect records from the aforementioned set of data including PII to create a set of token matches, and A cloud computing environment equipped with [features / equipment].

14. The cloud computing environment according to claim 13, further comprising a normalized UDF configured to normalize the set of data including PII.

15. The cloud computing environment according to claim 14, further comprising an index UDF configured to generate an index of prospect token combinations from the set of data including PII.

16. The cloud computing environment according to claim 15, wherein the index UDF is further configured to merge third-party matchable entities.

17. The cloud computing environment according to claim 16, wherein the index UDF is further configured to generate scored token permutations for the set of data including PII.

18. The cloud computing environment according to claim 17, wherein the matching logic UDF is further configured to filter the set of token matches to create a filtered set of token matches.

19. The cloud computing environment according to claim 18, wherein the matching logic UDF is further configured to create a set of matched customer records from the filtered set of token matches.

Images