Apparatus and method for probabilistic population size and overlap determination, remote processing of private data and probabilistic population size and overlap determination for three or more data sets

Abstract

The invention determines the population size and population overlap in data containing records on the unique entities without unique identifiers for the unique entities and having at least one common type of information with a known distribution of finite expectation by decomposing probabilistic calculations. The computer determines population overlap of unique entities between the data sets by subtracting a probabilistic incremental number of unique entities needed for a larger total number of values of the information with the known distribution from the data sets. The invention can also maintain the security of private data by allowing a remote computer where the original data is stored to download diagnostic and aggregation procedures from another computer over a network. The remote computer performs the functions on the data and forwards the results to the estimate processor computer over the network. The estimate processor determines population size and overlap from aggregate results and forwards this information back to the remote computer over the network. The invention also determines the overlap of three or more data sets by concatenating all combinations of the data sets and determining estimates for all subsets of the combinations of the data sets. The operations involve the cancellation of equivalent terms that have opposite signs.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS [0001] This application is a divisional of application Ser. No. 10 / 234,184 filed Sep. 5, 2002, which is a continuation-in-part of and claims priority to U.S. application entitled Apparatus And Method For Probabilistic Population Size And Overlap Determination, U.S. Pat. No. 6,470,298, by Banks and Pandiani, filed Feb. 4, 1997 both of which are incorporated by reference herein.REFERENCE TO COMPUTER PROGRAM LISTING, COMPACT DISK APPENDIX [0002] A compact disc is included in application Ser. No. 10 / 234,184 and incorporated by reference herein having thereon a computer program listing appendix (Appendix I) in the ASCII uncompressed text format with ASCII carriage return, ASCII line feed and all control codes defined in ASCII, having computer compatibility with IBM PC / XT / AT or compatibles, having operating system compatibility with MS-Windows and including file Appendix I of 54.6 kilobytes created on Sep. 4, 2002. BACKGROUND OF THE INVENTION [0003]...

AI Technical Summary

Benefits of technology

"The present invention provides an apparatus and method for accurately identifying the quantity of unique individuals in a data source containing multiple records pertaining to a single individual. This is achieved by using a computer to probabilistically calculate the population size of unique entities based on information with a known distribution of finite expectation. The invention also allows for the identification of unique individuals that overlap across multiple data sets. The invention can maintain the privacy of the data source and provide a data overlap processor that can determine the size and overlap of data sets containing records on unique entities. The technical effects of the invention include improved accuracy in identifying unique individuals and the ability to determine population size and overlap across multiple data sets."

Problems solved by technology

The measurement of population overlap has been hampered by both the complexity of the social institutions and the lack of unique personal identifiers across existing data sets.

Existing fragmented information systems could not support the critical epidemiological functions of determining the relationship among existing data sets.

First, the creation of case registries is a very expensive undertaking.

Second, the completeness of a registry is always in question, especially when participation is voluntary.

The incompleteness of the Gulf War Registry is notorious.

Finally, because they necessarily include personal identifiers, the creation of case registries raises important issues about personal privacy and confidentiality of personal records.

While the implementation of such identification systems has been successfully accomplished for specific organizations (e.g. individual hospitals, correctional facilities, and insurance companies), these identification systems do not constitute the kind of universal identification systems that allow for analysis of membership overlap.

In the United States, the social security number comes close to providing a universal identification system, but concerns about personal privacy severely limit the availability of these identifiers in settings not directly related to the social security system.

First, it is tedious, time consuming, and expensive.

Second, it includes an unquantifiable degree of error.

This approach also depends on personal identifiers, so concerns about privacy and confidentiality are likely to limit its utilization.

The problem of measuring the overlap between populations where no unique person identifier exists is related to the problem of measuring population size (the number of distinct individuals) without a unique person identifier.

It avoids the cost associated with complete case-by-case matching, but still raises issues of personal privacy and confidentiality because it relies on personal identifiers for a subset of the population.

In addition, his solution does not address the population overlap problem.

The difficulty of measuring the overlap between data sets tends to increase with the number of data sets involved.

This is because multiple data sets are less likely to share person identifiers, state and federal regulations are more likely to limit access to one or more of the data sets because of privacy concerns.

Images