Data Analytics Privacy Platform with Quantified Re-identification Risk

Abstract

The present invention is directed to a differential privacy platform in which the privacy risk of a computation can be objectively and quantitatively calculated. This measurement is performed by simulating high-performance privacy attacks against the system for various measures of privacy cost, or epsilon, and measuring the level of success of the attack. In some embodiments, a linear program reconstruction-type attack is used. By calculating the loss of privacy resulting from a sufficient attack at a particular epsilon, the platform can calculate the level of risk for a particular use of data. The privacy budget for the use of data can thereby be set and controlled by the platform to remain below a desired risk threshold.

Description

[Technical Field]

[0001] This application is filed in accordance with the 202 0 This application claims the benefit of U.S. Provisional Patent Application No. 63 / 080,333, filed September 18, 2003, which is incorporated herein by reference in its entirety. [Background technology]

[0002] Information barriers limit the use of data analytics. These information barriers can take many forms. Data privacy regulations, such as the General Data Protection Regulation (GDPR) in the European Union and the California Consumer Privacy Act (CCPA), restrict the access and movement of personal information. Similarly, organizations may be subject to myriad data privacy clauses that restrict the use of data as a condition of gaining access to it. The movement of data between locally hosted and cloud environments also poses barriers. Various private agreements or best practice restrictions may impose barriers to the movement of data for confidentiality reasons within an organization.

[0003] Some of the most highly protected private information is individual patient medical data. In the United States, such data is protected by, among other legal frameworks, the federal Health Insurance Portability and Accountability Act of 1996 ("HIPAA") and its implementing regulations. HIPAA provides fairly strict protections for various types of medical data and also provides very significant restrictions on the storage and transmission of this type of information.

[0004] While protecting private health data is of course essential, it is also true that analysis performed on medical data is critically important to advancing medical science and thereby improving the quality of health care. The COVID-19 pandemic provides a dramatic example, in which researchers' ability to analyze data related to COVID-19 patients and the various treatments provided to these patients has proven extremely important in clinicians' ability to provide improved care, leading to better outcomes for patients with this disease.

[0005] Under HIPAA, there are two ways to de-identify data so that it can be disclosed. The first is through "safe harbor," whereby 18 types of identifiers are removed from the data, including, for example, names, phone numbers, email addresses, and Social Security numbers. However, a team of researchers including Dr. Latanya Sweeney at Harvard University recently showed that this approach is not entirely sufficient to protect privacy against all forms of attack.

[0006] The second method for de-identifying data under HIPAA is the "Expert Determination" method. This requires a person with appropriate knowledge of, and experience with, generally accepted statistical and scientific principles and methods to determine that there is an extremely small risk that the information could be used by the intended recipient, alone or in combination with other reasonably available information, to identify the individual to whom the information relates. The "risk-based de-identification" concept within the GDPR is similar to the Expert Determination method under HIPAA. However, there is no explicit numerical level of de-identification risk that is universally considered to meet the "extremely small" level indicated by that method.

[0007] Generally, there are several principles that experts must consider in their expert determination. One is replicability, i.e., the risk that data will be consistent for an individual. For example, a patient's blood glucose levels fluctuate and therefore have low replicability. On the other hand, a patient's birthdate has high replicability. The second principle is data source availability, i.e., how likely the information is to be available in a public or other accessible source. For example, laboratory report results are rarely disclosed with identifying information outside of a healthcare setting, but names and demographic data often are. The third principle is distinguishability, i.e., how unique the information can be for an individual. For example, the combination of birth year, gender, and three-digit ZIP code is unique for only a small number of U.S. residents, but the combination of birth date, gender, and five-digit ZIP code is unique for more than 50% of U.S. residents. The fourth principle is risk assessment, which combines these other principles into an overall analysis. For example, laboratory results can be highly specific, but they are rarely disclosed in multiple data sources to which many people have access, whereas demographics are highly specific, highly repeatable, and available in public data sources.

[0008] The use of "very small" as a measure of risk under HIPAA recognizes that the risk of re-identification in a database is never zero. If data has any utility, there is always some risk that the data may be re-identified, although it may be insignificantly small. It is also known that the lower the privacy risk, the lower the data's utility, because de-identification to a certain point can render the data seldom or never usable for its intended purpose. Generally, a 50% re-identification risk is said to be at risk of re-identification. Meanwhile, lowering the re-identification risk to 0.05%–0.10% is generally considered acceptable. However, the challenge is determining the actual re-identification risk in a particular instance.

[0009] Therefore, privacy protection through expert determinations can prove to be highly subjective and variable. It would be desirable to provide a more solid mathematical basis for determining the level of risk so that risk can be assessed in an objective manner, both to determine the risk in a particular scenario and to compare the risks posed by different scenarios.

[0010] Differential privacy is a method of protecting privacy based on the principle that privacy is a property of a computation on a database, as opposed to the syntactic qualities of the database itself. Essentially, a computation is considered differentially private if it produces nearly the same results when applied to two databases that differ only by the presence or absence of a single data subject's record. It will be understood that the level of differential privacy for a particular computation will depend greatly on the data at issue. For example, if a computation is performed on average income and the income of an individual called John Doe is close to the average for the entire dataset, the results will be nearly the same regardless of whether John Doe's data is removed, and the privacy loss will be low. On the other hand, if John Doe's income is much greater than others in the dataset, the results may be significantly different, and the privacy loss for this computation will be high.

[0011] Differential privacy is powerful because of the mathematical and quantifiable guarantees it provides regarding the re-identifiability of the underlying data. Differential privacy differs from historical approaches because of its ability to quantify the mathematical risk of de-identification. Differential privacy makes it possible to track the cumulative privacy risk for a dataset across many analyses and queries.

[0012] As of 2021, over 120 countries have laws governing data security. As a result, compliance with all of these regulatory regimes can seem impossible. However, most data security regulations, such as GDPR, CCPA, and HIPAA, are unified around the concept of data anonymization. The problem, however, is that differential privacy techniques, as they are known before, do not map well to the concepts and anonymization protocols presented in these various privacy laws. For this reason, differential privacy has experienced very limited adoption despite its great promise.

[0013] Historically, differential privacy research has focused on theoretical problems in which an attacker has access to all possible information related to a data set other than the specific item of data sought to be protected. However, the inventors of the present application have recognized that this all-knowing adversary is not a realistic model for determining privacy and does not address the rationality requirements under the HIPAA and GDPR protection requirements outlined above. These regulations address real-world privacy situations, not highly theoretical situations in which an attacker has all possible information.

[0014] Furthermore, to bring differential privacy to practical applications and within the framework of existing privacy regulations, it is necessary to determine what level of epsilon (i.e., the privacy "cost" of a query) provides reasonable protection. Current work on differential privacy simply selects and applies a particular epsilon without providing support for why that epsilon was chosen or why that particular choice of epsilon provides sufficient protection under any of the various privacy regulations, such as HIPAA and GDPR. To use differential privacy in practical applications under these existing legal frameworks, a method for quantifying privacy risks that fits within such frameworks is needed. [Prior art documents] [Non-patent literature]

[0015] [Non-Patent Document 1] Cohen et al., "Linear Program Reconstruction in Practice," arXiv:1810.05692v2[cs.CR], January 23, 2019. Summary of the Invention

[0016] The present invention is directed to a differential privacy platform in which the privacy risk of a calculation can be objectively and quantitatively calculated. This measurement is performed by simulating high-performance privacy attacks against the system for various measures of privacy cost, or epsilon, and measuring the level of success of the attack. In some embodiments, a linear program reconstruction attack is used as an example of one of the most high-performance types of privacy attacks. By calculating the loss of privacy resulting from an attack at a particular epsilon, the platform can calculate the level of risk for a particular use of the data. A privacy "budget" for the use of the data can thereby be set and controlled by the platform to remain below a desired risk threshold. By maintaining privacy risk below a known threshold, the platform provides compliance with applicable privacy regulations.

[0017] In various embodiments, the present invention uses differential privacy to protect the confidentiality of individual patient data. To protect patient privacy while enabling the most valuable uses of the data, the present invention in various embodiments provides an objective, quantifiable measure of the re-identification risk associated with a particular use of the data, thereby ensuring that no significant risk of re-identification occurs within a proposed data analysis scenario.

[0018] In various embodiments, raw data is not exposed or moved outside its original location, thereby providing compliance with data privacy and localization laws and regulations. In some embodiments, the platform can anonymize validated models for privacy and compliance purposes, and users can export and deploy secure models outside the original data location.

[0019] In some embodiments, the computing platform can generate differentially private synthetic data that represents the underlying dataset, allowing data scientists and engineers to assemble data prep, data cleaning, and feature pipelines without ever seeing the raw data, thereby preserving privacy.

[0020] In some embodiments, familiar libraries and frameworks such as SQL can be used by data scientists to define machine learning models and queries. Users can interact with the platform according to some embodiments by submitting simple commands using specific APIs.

[0021] In some embodiments, the present invention uses a metric to assess privacy risk from intentional attacks, where the probability of a successful privacy attack, Pr(Success), is equal to the probability of success if the attack is performed, Pr(Success|Attempts), multiplied by the probability of the attack, Pr(Attempts). The present invention then provides the adversary model described above that represents the most significant risk of attempted privacy attacks, summarizes the mitigation controls in the consortium, and presents the determined Pr(Attempts) given consideration of these factors. Industry best practices provide a reference point for deriving the Pr(Attempts) value. When strong security protocols, including multi-factor authentication, HTTPS, etc., are present in the system, Pr(Attempts) is typically set between 0.1 and 0.25. A strong privacy attack is used to calculate Pr(Success[]Attempts). Once these two values ​​are known, Pr(Success) can be calculated as the product of these two values.

[0022] In some embodiments, the present invention may employ caching to return the same noisy results in a differential privacy implementation regardless of the number of times the same query is submitted by a particular user. This caching may be used to thwart some types of privacy attacks that attempt to filter out noise by averaging results.

[0023] In various embodiments, the platform deploys differential privacy as an enterprise-wide distributed system. While an enterprise may have hundreds, thousands, or even tens of thousands of data stores, the platform provides a unified data layer that allows analysts to interact with data regardless of where or how it is stored. The platform provides a privacy ledger to ensure mathematical privacy across all connected datasets through a unified interface. The platform also has a rich permission layer that enables authorization based on user and data attributes. The platform allows control over who can run queries and with what type of privacy budget.

[0024] These and other features, objects and advantages of the present invention will become better understood upon consideration of the following detailed description of the preferred embodiments and the appended claims, taken in conjunction with the drawings, as set forth below. [Brief explanation of the drawings]

[0025] [Figure 1] 1 is a chart illustrating the level of risk for re-identification of data based on data protection schemes. [Figure 2] FIG. 1 illustrates the concept of differential privacy. [Figure 3] 1 is a flow chart for a system according to one embodiment of the present invention. [Figure 4] FIG. 2 is a swim lane diagram for a system according to one embodiment of the present invention. [Figure 5] FIG. 10 is a swim lane diagram for an external researcher SQL query, according to one embodiment of the present invention. [Figure 6] FIG. 10 is a swim lane diagram for an internal investigator SQL query, according to one embodiment of the present invention. [Figure 7] FIG. 1 is a swim lane diagram for external researcher machine learning training or evaluation, according to one embodiment of the present invention. [Figure 8] FIG. 1 is a swim lane diagram for internal researcher machine learning training or assessment, according to one embodiment of the present invention. [Figure 9] FIG. 10 is a swim lane diagram for external researcher synthetic data queries, according to one embodiment of the present invention. [Figure 10] FIG. 10 is a swim lane diagram for internal researcher raw data queries, according to one embodiment of the present invention. [Figure 11] FIG. 1 is a high-level architectural diagram of a data environment according to one embodiment of the present invention. [Figure 12] FIG. 10 illustrates an exemplary SQL query and results that expose raw data, if enabled, according to one embodiment of the present invention. [Figure 13] FIG. 10 illustrates an exemplary SQL query for average length of stay, according to one embodiment of the present invention. [Figure 14] FIG. 10 illustrates an exemplary SQL query in an attempt to manipulate the system to expose private information, according to one embodiment of the present invention. [Figure 15] FIG. 1 illustrates an exemplary SQL query with added noise, according to one embodiment of the present invention. [Figure 16] FIG. 1 illustrates an exemplary SQL query attempt for identifying private data using quasi-identifiers, according to one embodiment of the present invention. [Figure 17] FIG. 10 illustrates an exemplary SQL query for patient counts binned by date range of death, according to one embodiment of the present invention. [Figure 18]FIG. 2 illustrates an exemplary query for creating a synthetic dataset, according to one embodiment of the present invention. [Figure 19] FIG. 2 illustrates an exemplary query for performing machine learning analysis, according to one embodiment of the present invention. [Figure 20] 1 is a chart providing a summary of an exemplary configuration for a data analytics platform, according to one embodiment of the present invention. [Figure 21] 1 is a graphic providing an example of a query that consumes a query epsilon budget, according to one embodiment of the present invention. [Figure 22] FIG. 1 illustrates an SQL query by a prior art system to perform a successful differential attack on a database. [Figure 23] FIG. 22 illustrates that the SQL query of FIG. 21 is thwarted by differential privacy, according to one embodiment of the present invention. [Figure 24] 10 is a chart illustrating the results of a privacy attack at varying values ​​of epsilon per query, according to one embodiment of the present invention. [Figure 25] 10 is a graph plotting the results of a privacy attack at varying values ​​of epsilon per query, according to one embodiment of the present invention. [Figure 26] FIG. 1 illustrates an SQL query by a prior art system to perform a successful averaging attack on a database. [Figure 27] 27 is a chart showing the results of the averaging attack of FIG. 26. [Figure 28] 27 is a chart showing the results of the averaging attack of FIGS. 25 and 26 on one embodiment of the present invention with caching. [Figure 29] 1 is a graphic illustrating an exemplary linear programming reconstruction attack on a database. [Figure 30] FIG. 10 illustrates the results of a reconstruction attack on an embodiment of the present invention at varying levels of total epsilon. [Figure 31]10 is a density chart showing the results of a reconstruction attack on one embodiment of the present invention at varying levels of total epsilon; [Figure 32] 1 is a chart illustrating parameters for an exemplary reconstruction attack for one embodiment of the present invention. [Figure 33] 10 is a chart illustrating the results of an attribute inference attack on one embodiment of the present invention. [Figure 34] 1 is a chart illustrating disclosure risk resulting from a synthetic dataset, according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0026] Before the present invention is described in further detail, it should be understood that the invention is not limited to the particular examples described, and that the terminology used in describing particular examples is for the purpose of describing those particular examples only, and not as limiting, since the scope of the invention is limited only by the claims. In particular, although the invention in various embodiments is described with respect to the use of protected health information in various scenarios, the invention is not so limited and may be employed in alternative embodiments with respect to any type of data for which data privacy is to be protected.

[0027] Re-identification risk for data including personal health data can be roughly divided into five levels or categories, as shown in Figure 1. In this example, the risk is described with respect to medical data protected by HIPAA in the United States, although the invention is not so limited. Data at higher levels has less risk of re-identification, but requires more effort, cost, skill, and time to re-identify for that level.

[0028] At Level 1 is readily identifiable data, i.e., raw data containing personal health information. In other words, this is full protected health information (PHI) with identifiers preserved. This type of data can be used to achieve the highest analytical utility for researchers, but its use also represents the greatest disclosure risk for patients.

[0029] At level 2 is masked data. This data still contains personal data, but it has been masked in some fashion. For example, there is some transformation of the personal information, such as placing the data into bands or ranges. This could include age ranges or larger geographic areas, such as by ZIP code. Masked data can also include data where things like demographics or other identifiers have simply been removed. Both the Safe Harbor provision and the Limited Data Set provision of HIPAA laws and regulations refer to masked data. Masking techniques can be either reversible or irreversible. Regardless, this data still contains indirect identifiers that pose a re-identification risk.

[0030] At level 3 is exposed data. This is data that has had privacy transformations applied, but that lacks rigorous analysis of re-identification risks. The risk associated with disclosing this data is difficult to quantify.

[0031] At level 4 is controlled data. This is data for which verifiable claims have been made that involve risk assessments based on rigorous methodologies. Controlled data may be identifiable above or below a certain threshold of privacy protection. Above this threshold, the data may still be considered to contain personal information, but below the threshold, it may be considered not to contain personal information.

[0032] At the highest level, Level 5, is data that appears only in aggregated form, i.e., by combining data about multiple data subjects that does not contain personal information. For example, the mortality rate for a cohort of patients is the aggregate of the counts of the cohort of individual survivors divided by the total cohort size. Aggregated data stratified by quasi-identifiers can be re-identified through privacy attacks; therefore, in these cases, rigorous analysis must be conducted to determine whether the data can be considered aggregated data. True aggregated data does not present a privacy risk because it cannot be re-identified by anyone.

[0033] Additional groupings can be applied to data types in this tier-based model: Levels 1-4 all represent "person-level" data, where each row in the dataset represents information about an individual. Level 5 is unique among these tiers in that it always represents information about a group of individuals and is therefore not considered PHI.

[0034] Aggregated data is not personal health information and therefore does not require de-identification risk management. Therefore, it is by definition properly de-identified. However, care must be taken when presenting aggregated data for research use, because the data presented is effectively aggregated data and cannot be reformatted or manipulated to expose individual health information. For example, if quasi-identifiers stratify aggregated information, privacy mechanisms must be employed to verify that the aggregated data cannot be used to re-identify individuals. In some embodiments of the present invention, all statistical results of a query are aggregated data, but because users can query results stratified by quasi-identifiers, the results present a privacy risk to the system. However, the present invention provides a means for explicitly assessing this privacy risk, as described below.

[0035] Differential privacy is based on a definition of privacy that asserts that privacy is a property of a computation on a database, as opposed to a syntactic quality of the database itself. In general, a computation is differentially private if it produces nearly the same results when applied to two databases that differ only by the presence or absence of a single data subject's record. Figure 2 provides an illustration of differential privacy with respect to data about a person named John Doe 10. A computation is differentially private if and only if it produces nearly the same query results 16 when applied to two databases 12 and 14 that differ only by the presence or absence of a single data subject's record.

[0036] This definition of differential privacy can be formally explained via a mathematical definition. For example, take a database D, which is a collection of data elements drawn from a universe U. Rows in the database correspond to individuals whose privacy needs to be preserved. Each data row has a set of attributes A=A1, A2, ..., A mThe set of values ​​that each attribute can take, i.e., their attribute domain, is dom(A i ) where 1≦i≦m. The mechanism M:D→R d is a randomizing function that maps a database D to a probability distribution over a range and returns a vector of randomly chosen real numbers within that range. A mechanism M is said to be (ε,δ)-differentially private if adding or removing a single data item in the database only affects the probability of any outcome within a small multiplicative factor exp(ε), except for sets whose densities exceed that boundary by a total of δ or less.

[0037] The sensitivity of a query function f represents the largest change in the output for the query function that can be made by a single data item. The sensitivity of a function f, denoted as Δf, is Δf=max|f(x)-f(y)| where the maximum is over all pairs of data sets x and y that differ by at most one data subject.

[0038] A differentially private mechanism can be implemented by introducing noise sampled from a Gaussian distribution. Specifically, the Gaussian mechanism adds noise sampled from a Gaussian distribution whose variance is chosen according to the sensitivity Δf and the privacy parameters ε and δ.

number

[0039] A key observation of differential privacy mechanisms is that the variance of the distribution from which the noise is sampled is proportional to Δf, ε, and δ. Importantly, this differs from other perturbation methods, which sample from the same distribution for all noise perturbations. Bounded-noise methods are vulnerable to attacks that decipher the noise parameters via differential and averaging attacks, then leverage this information to dynamically remove the noise and use the correct values ​​to re-identify individuals and / or reconstruct the database. For the same differential privacy mechanism applied to two subsets of the database, D1 and D2 ⊂ D, with equivalent privacy parameters ε and δ, the variance σ is proportional to the sensitivity of the function Δf computed for D1 and D2, respectively. This property increases the level of skill, time, and resources required to decipher private information from differentially private results compared to statistical results released under bounded-noise methods.

[0040] Each differential private query q executed by a user of the system is executed with user-configurable parameters, and therefore each submitted query is ε,δ Each of these queries consumes a privacy budget with values ​​ε and δ. The higher the ε and δ parameters of a query, the more accurate the results will be but the lower the privacy guarantees, and vice versa. Furthermore, each query reduces the total budget of the dataset by its configured ε and δ values.

[0041] As mentioned above, differential privacy is the only formal definition of privacy, and it is widely accepted in the research literature as a rigorous definition of privacy. However, its use in practice is limited by difficult problems that stem in part from choosing appropriate ε and δ parameters for a privacy budget. There is no formal model for selecting appropriate ε and δ parameters in a privacy system. Choosing values ​​that are too low degrades analytical utility to the point where the system cannot serve its intended function, while choosing values ​​that are too high can lead to catastrophic privacy loss.

[0042] The problem of setting a privacy budget has received less attention in the research literature than the differential privacy mechanism itself. Different approaches have been proposed, including economic and heuristic methods. While the information privacy literature for health data suggests that some systems have budgets as high as 200,000, differential privacy practitioners have suggested values ​​as low as less than 1 and as low as 100. 10 These wide ranges do not provide a means for applying differential privacy while operating in real-world environments under applicable legal regulations such as HIPAA.

[0043] Before describing the structure and operation of a platform for providing access to sensitive data with quantified privacy risks according to some embodiments of the present invention, the system's function within the overall ecosystem of medical research and publishing can be described. Several health providers have developed extremely large, high-fidelity, and high-dimensional patient datasets that are of great value for medical research. If health providers can form consortia to share their data for medical research while still complying with privacy requirements, they can leverage this data to generate even greater returns on their medical research and improved patient outcomes. A typical research consortium according to some embodiments of the present invention consists of four types of entities: research organizations, medical research journals, data providers (such as healthcare providers), and data analytics platform providers. In this configuration, research organizations enter into an agreement to collaborate to answer important research questions. Medical research journals receive articles describing the results of these studies. Medical journals can also be provided with data to corroborate the underlying data supporting these studies to avoid issues of falsified data. Data providers provide both their data assets and potentially their research personnel. The data analytics platform provider will use its platform to de-identify data in a secure environment so that researchers within the consortium can conduct analyses without exposing protected health information (PHI). Thus, the overall system protects patient privacy while enabling the most advantageous use of health information from data providers.

[0044] A basic workflow within a consortium, according to one embodiment of the present invention, can now be described with reference to Figure 3. In step 30, a principal investigator at a research organization proposes a research study. The research organization has established a data use agreement with a data provider for this purpose.

[0045] In step 32, a central institutional review board reviews the proposed studies and either approves or rejects them. If approved, the board assigns appropriate access tiers for the investigators involved in the studies. All of the studies are conducted in a secure cloud infrastructure powered by the data analysis platform described herein.

[0046] In step 34, an analyst (who, in some embodiments, is a researcher) uses the data analysis platform to conduct research that complies with all privacy requirements through the analyst's own computer system, which is networked to the data analysis platform. The data is de-identified before it reaches the researcher. Access to the data is provisioned according to two different roles: internal researchers, who can access PHI, and external researchers, who can only receive de-identified information. Internal researchers are researchers associated with the data provider and are given access that complies with HIPAA's "limited data set" standard. For example, patient data available to internal researchers may include unlimited patient ages and five-digit ZIP codes. For external users, three types of usage patterns are available. The first is SQL queries through an interactive API, which provides limited access to secure data in the cloud infrastructure. Results from this API call are perturbed with noise to preserve privacy using differential privacy techniques. The second usage pattern is machine learning. A Keras-based machine learning API is available, which allows for fitting and evaluation of machine learning models without ever removing the data from the secure cloud infrastructure. The third usage pattern is synthetic data. This data is artificial, but statistically equivalent to the original data, and is a computational derivative of the original data. Synthetic data does not contain PHI. In alternative embodiments of the invention, only one or two of these functions may be available.

[0047] Please note that the expert determination under HIPAA described herein only addresses de-identification methods applicable to the External Investigator Permission Tier. HIPAA compliance of data use by research organizations operating within the Internal Investigator Tier may be determined based on the data used and the executed agreement between the data provider and the research organization. However, there are points in the research workflow where external and internal investigators may collaborate. Controls are put in place to ensure that external investigators do not access protected health information during the process.

[0048] Once they are assigned the appropriate data access tier, researchers use the data analytics platform to conduct research and analysis on the data provider's datasets. All data access occurs through an enclave data environment that protects the privacy and security of the data provider's data. Firewalls and other appropriate hardware are employed for this purpose. The data analytics platform is a privacy-preserving computing framework throughout the data science lifecycle.

[0049] Before researchers are granted access to the data analysis platform, the system is installed and configured within the data provider's cloud infrastructure. This setup process includes configuring privacy parameters, which is described below. Another stage in system setup is configuring security safeguards to ensure that only authorized users are granted access to the system. While protecting data involves both security and privacy protection, the focus of this discussion is privacy, and the system may also use various security mechanisms, such as multi-factor authentication, to provide security.

[0050] External researchers can create SQL views and run SQL queries through the data analytics platform API. Queries run through the SQL API return approximate results protected by differential privacy, which means that a controlled amount of noise is injected into the results to protect the privacy of data subjects. However, researchers can use these approximate results to explore the data provider's dataset and develop research hypotheses. Once a researcher has a hypothesis and requires exact values ​​to be returned, the researcher sends an analysis to the internal researcher. This internal researcher can run the analysis, extract the exact results, and use them to provide insights to the external researcher. The only results the internal researcher can provide to the external researcher are aggregate statistical results. The internal researcher is responsible for verifying that the information sent to the external researcher does not violate the privacy of individual patients. Because multiple seemingly innocuous queries can be used together to reveal sensitive information, the internal researcher must be aware of the context and purpose of the analysis conducted by such an external researcher.

[0051] In step 36, the article is written and reviewed for disclosure prior to publication. This review is to ensure that PHI is not inadvertently disclosed in the article. In step 38, the article is submitted for publication to one or more medical research journals. These may be accessed through specific resource centers for specific health concerns.

[0052] The workflow for this just-described process is provided in the swim lane diagram of Figure 4. In step 40, the external researcher submits a study from their computer system and applies for access to the data to an internal review board. In step 41, the board reviews the proposal and either approves or rejects it. If the proposal is approved, the process moves to steps 42 and 43, where the external researcher and internal researcher, respectively, are given access to the data analysis platform for the purpose of the study. The external researcher, in step 44, runs queries and generates hypotheses about the noisy returned data. In step 45, the external researcher contacts the internal researcher to test the hypotheses against a non-noisy (i.e., raw) version of the data. The internal researcher, in step 46, evaluates the hypotheses as requested and determines, in step 47, what results can be returned to the external researcher while maintaining appropriate privacy safeguards. In step 48, the internal investigator returns the non-disclosive aggregated statistical results of the hypothesis evaluation to the external investigator, and they may review the results together in step 49. If the results are interesting or important, such as a confirmation of the external investigator's hypothesis, the investigator selects the data for inclusion for publication in step 50. This data is sent to a committee for review in step 51, and if approved, the investigator then prepares a manuscript for publication in step 52. The investigator submits the manuscript for publication to a medical journal(s) in step 53, and the public then gains access to the article upon publication in step 54. The article may include aggregated results because such results do not disclose PHI.

[0053] Figure 5 details the workflow for SQL API queries for an external researcher. In step 60, the external researcher submits an SQL query related to the data provider's dataset through the API. In step 61, the data analytics platform runs the query against the raw dataset but injects noise into the results as part of the differential privacy scheme. In step 62, the noisy results are returned to the external researcher, and in step 63, the external researcher receives the noisy results.

[0054] Figure 6 details the workflow for an SQL API query for an internal researcher. Similar to Figure 5, a query is received through the API in step 70. However, in this case, the data analytics platform runs the query against the raw dataset without injecting noise in step 71. The true results are returned in step 72, and then the internal researcher receives the true query results through the API in step 73.

[0055] As described above, external researchers can train and evaluate machine learning models on SQL views within the data analytics platform. These models are defined through the Keras API and trained and evaluated remotely on the data provider's clinical dataset. The models themselves are not returned to the researcher; evaluation can only be performed using data residing within the secure cloud infrastructure environment. Figure 7 provides a flow for this process. In step 80, the external researcher requests machine learning training or evaluation through the corresponding API. In step 81, the data analytics platform ingests the raw data and performs the requested machine learning task. In step 82, the platform returns status and / or summary statistics, if applicable, to the researcher. In step 83, the external researcher receives the status and / or summary statistics through the API.

[0056] Internal researchers may also access and export models created by external researchers, but these models tend to remember the characteristics of the data on which they are trained and are therefore treated within the system as if they were private data. Figure 8 provides a detailed flow for this process. In step 90, the internal researcher requests training or evaluation through an appropriate API. In step 91, the data analytics platform ingests the raw data and performs the requested machine learning task. In step 92, the platform returns status and / or summary statistics, if applicable, to the internal researcher. The internal researcher receives the status and / or summary statistics in step 93. The internal researcher may then request the trained machine learning model through the API in step 94, and the data analytics platform retrieves and returns the trained machine learning model in response to this request in step 95. The internal researcher then receives the trained machine learning model through the API in step 96.

[0057] Again, as mentioned above, synthetic data can also be used in this process. An external researcher can create and export synthetic datasets generated from SQL views based on actual datasets. These synthetic datasets retain some of the univariate and multivariate statistical properties of the original datasets on which they are based, and therefore can be used to generate research hypotheses. For example, an external researcher can use the synthetic dataset to prepare scripts that run regressions or hypothesis tests. FIG. 9 provides a flow for this process. In step 100, the external researcher requests the synthetic dataset through a corresponding API. In step 101, the data analysis platform generates the synthetic dataset, and in step 102, the data analysis platform evaluates the privacy of the synthetic dataset before its release. If the synthetic dataset is sufficiently private, the data analysis platform releases the synthetic dataset, which is received by the external researcher through the corresponding API in step 103.

[0058] Internal researchers may also use synthetic datasets. Just as with SQL workflows, internal researchers can run analyses to extract rigorous results and use those results to provide insights to external researchers. As mentioned above, the only information an internal researcher can send to an external researcher is aggregate statistical results. This flow is shown in FIG. 10. In step 110, the internal researcher requests the raw dataset through a corresponding API. The data analysis platform retrieves the raw dataset in step 111, and then the internal researcher receives the raw data from the data analysis platform through a corresponding API in step 112.

[0059] Now that this overall system description is complete, a system and method by which de-identification is performed within a data provider's dataset in accordance with HIPAA or other applicable privacy rules can be described in more detail. In the illustrative example provided herein, the data provider's dataset is a relational database of inpatient and intensive care unit (ICU) encounters that maintains various data fields that are not compliant with HIPAA safe harbors, such as year of birth (or age) and date of service. These fields are maintained to allow epidemiological research studies to be conducted on the data. Due to the presence of identifying and semi-identifying fields, the data must be de-identified via expert determination under HIPAA rules. The expert determination relies on the application of statistical or scientific principles that create only a "very small" risk that an individual could be identified.

[0060] The de-identification systems and methods described herein operate according to three core principles. The first principle is that there is no row-level access to PHI. This means that analysts are never exposed to PHI. The entire analytical lifecycle, from data transformation to statistical analysis, is supported without compromising row-level protected health information.

[0061] The second principle is the use of noise to prevent unique disclosures. Aggregate data is created using rigorous statistical privacy techniques to reduce the risk of leaking sensitive information. Differential privacy, as described herein, reinforces this capability.

[0062] The third principle is that PHI remains secure by enforcing policies for granular permissions that grant access to analysts without publicly releasing the data or ever moving the data outside the data provider's firewall.

[0063] The de-identification system and method are first described by providing a high-level summary of the privacy mechanisms employed in the data analytics platform. The second section provides a summary of privacy-related considerations and parameters selected to achieve a HIPAA-compliant "negligible" risk that individuals could be re-identified. The third section provides a quantitative assessment of the privacy risks presented by the system and method.

[0064] The data analytics platform is implemented as a cluster computation framework that enables analysts and data scientists to perform data transformations, feature engineering, exploratory data analysis, and machine learning, all while maintaining patient privacy on the underlying data. All data access in the data analytics platform is through an enclave data environment, which protects the privacy and security of the data provider's data. The data platform provides controls to ensure that data cannot leave the data provider's cloud environment where it is hosted. Figure 11 provides a high-level diagram of this environment. Both the data analytics platform 114 and the data provider dataset 116 reside within the enclave data environment 118. An analyst 119 (operating through a remote computing device connected to the enclave data environment 118 via a network such as the Internet) can access the data provider dataset 116 only through the data analytics platform 114, thereby ensuring that the analyst (e.g., an external researcher) never sees row-level data within the data provider dataset 116.

[0065] The data analytics platform 114 performs three core analytics functions: executing SQL queries through an SQL API, developing machine learning models through a machine learning API, and generating synthetic datasets through a synthetic dataset API. Each of these three functions has safeguards to protect the privacy of patients whose data resides in the data provider datasets 116. While typical de-identification assessments may require attribute-level privacy assessments, in the case of the data analytics platform 114, privacy is enforced by system mechanisms equally across all attributes. Thus, the privacy controls described herein remain in effect even if additional attributes, such as a ZIP code field, are added to the data provider datasets 116.

[0066] Researchers interact with the data analytics platform 114 much as they would interact directly with a database. However, in this case, the data remains within the data provider's cloud infrastructure environment, i.e., the enclave data environment 118. Standard SQL syntax, familiar to many researchers, can be used for the necessary API calls. Figure 12 provides an example of an SQL query that would expose raw data from the data provider dataset 116 if it were enabled; however, as shown in Figure 12, an error occurs because this type of query is rejected by the data analytics platform 114. Executing this "*" query would result in a dump of all patient information, thereby causing a catastrophic loss of privacy.

[0067] Although some query restrictions are imposed to protect patient privacy, the data analytics platform 114 supports aggregate statistical queries in standard SQL syntax. Figure 13 shows how an analyst could use such a command to query a table for the average length of stay across patient encounters in a dataset. While this type of query is allowed, it adds noise to the data, as explained more fully below.

[0068] Some aggregate queries can be manipulated to expose private information. For example, a user may attempt to single out a specific user via an identifier. Figure 14 shows a query that would single out information about a single patient and illustrates how the data analytics platform 114 prevents this behavior.

[0069] Simple protection of the type shown in FIG. 14 guards against most malicious attempts against the system to expose private information. However, malicious users motivated to extract sensitive information from the dataset can launch more sophisticated privacy attacks against the system in an attempt to exfiltrate sensitive private data. To this end, the data analysis platform 114 employs an additional layer of protection based on differential privacy, which adds noise to the output of statistical queries. The noise is added in a controlled manner to minimize the impact on analysis quality, as shown in FIG. 15. In this case, the noise is added to the true mean age of 58.3, and the returned value is 60.1. While this data is still useful to external researchers for forming hypotheses, the noise thwarts many types of privacy attacks that attempt to re-identify data in the data provider dataset 116.

[0070] Some queries provide result sets that stratify statistical results by quasi-identifiers, and this information can result in unique disclosure of protected health information. For example, a query returning the number of deaths by day can uniquely disclose private information. The data analytics platform 114 dynamically prevents disclosing results from being returned to the analyst. An example of this is shown in FIG. 16, where an attempt is made to return a count by date of death. As shown in FIG. 16, an error is returned and no data is retrieved.

[0071] Analysts can manipulate queries like those shown in Figure 16 to return similar types of information, but with results of lower fidelity. For example, for the same dataset, analysts can query for the number of deaths by week (instead of by day), and the differential privacy mechanism dynamically calculates whether the result is revealing. If the binned values ​​are not revealing, they are returned to the analyst, and each week has a carefully calculated amount of noise added to maximize statistical utility for the analyst while preserving the privacy of data subjects in the database. The method for determining whether a binned value is revealing depends on the privacy budget, as explained below.

[0072] Differential privacy protects datasets by providing mathematical guarantees regarding the maximum influence a single individual can have on the output of a process, such as a query. The system is designed around differential privacy techniques without strictly adhering to the formal definition required for a strong adversary (e.g., a theoretical adversary with possession of all possible useful information). In particular, the data analysis platform 114 employs an empirical risk assessment of the system, as opposed to utilizing theoretical risk values ​​produced by academic researchers' privacy mechanisms. While this forfeits the theoretical mathematical guarantees associated with a pure implementation of differential privacy, this implementation is quantitatively shown to adequately protect confidentiality as well as anonymity. An example query in this category is shown in Figure 17, where counts of death dates are binned by week.

[0073] As mentioned above, another core function of the data analytics platform 114 is enabling synthetic data generation. This capability allows users to generate artificial datasets that maintain similarity to the PHI dataset from which they were generated. An example query for creating a synthetic dataset is provided in FIG. 18. Synthetic data is generated using machine learning models, and the synthesized data is assessed for privacy risks before being returned to the user. The synthetic data appears structurally similar to the data from which it was generated and can be used by any data analytics toolkit, such as the scientific computing library of the Python programming language or the R programming language and its many libraries for data analysis.

[0074] The third core function of the data analytics platform 114 is machine learning. Machine learning models present a privacy risk because they can memorize sensitive data as they are trained. To mitigate this risk, the system allows analysts to develop and evaluate machine learning models without ever having direct access to the underlying protected health information. Figure 19 shows how an analyst can develop a machine learning model using the machine learning API associated with the data analytics platform 114. The researcher sets up Keras as the tool in which the model will be developed, defines the model using a template, and commands the data analytics platform 114 to train the model using the desired data, focusing on the health question underlying the researcher's specific hypothesis. The output includes a graph providing an analysis of sensitivity and specificity via the area under the receiver operating characteristic (ROC) curve. Researchers can evaluate models using standard regression and classification metrics, including the use of confusion table metrics, but cannot extract the actual trained model or its coefficients because they may leak private data.

[0075] The data analytics platform 114 has several parameters that must be configured prior to use. These ensure effective protection of PHI (or, in non-medical applications, other types of private information). In one illustrative example, the settings are summarized in the chart in FIG. 20. In this example, the re-identification risk is set very low at a conservative value of 0.05 percent. The overall privacy "budget" is set with an epsilon value per table of 1000 and an epsilon value per query budget of 1. The "likelihood constant for privacy attack probability" is a measure of the likelihood that any user with security access to the system will actually engage in an attempt to circumvent the privacy safeguards in the system. Each of these issues is explained more fully below.

[0076] The re-identification risk threshold refers to the maximum acceptable risk that an individual may be re-identified in a dataset. Risk values ​​below the threshold are considered "very small" risk. There are many statistical methodologies for assessing re-identification risk that measure different types of privacy risk. One methodology for assessing re-identification risk focuses on quantifying the acceptable risk value for data released using the HIPAA safe harbor and demonstrating that the expert decision method presents less re-identification risk to patients than the safe harbor approach. In one survey of re-identification risk under the safe harbor, researchers analyzed what percentage of unique disclosures would be permitted in data released under the safe harbor. Researchers found that a 4% uniqueness of individuals is acceptable under the safe harbor, indicating that a group containing only four individuals could be reported in de-identified data.

[0077] Another methodology focuses on closely interpreting guidance from the Department of Health and Human Services (HHS). HHS hosts guidance from the Centers for Medicare and Medicaid Services (CMS) on its website. The guidance states that "cells containing values ​​between 1 and 10 (e.g., admission, discharge, patient, service, etc.) cannot be reported directly." This guidance has been interpreted by the European Medicines Agency (EMA) to mean that only groups containing 11 or more individuals can be reported in de-identified data, leading to the partial adoption of 0.09 (or 1 / 11) as the maximum acceptable risk presented to a single individual. This is not to say that the EMA interpretation applies to the United States, but it is not irrelevant to determining what an acceptable amount of privacy risk is for de-identified data. These same methodologies may apply under different regulatory regimes.

[0078] Both of the above methodologies measure the maximum risk of unique disclosure associated with an individual record. Another methodology is to measure the average re-identification risk across all individual records in the entire de-identified database. For both the average risk and maximum risk approaches, a conservative risk threshold is less than 0.1. One study found that the safe harbor method of de-identification leads to a risk of approximately 0.04, meaning that nearly 4% of patients in a safe harbor de-identified database are re-identifiable. The upper end of an acceptable risk is closer to 0.5, which means the requirement is not to uniquely disclose individuals, and therefore groups as small as two individuals may be disclosed in a data release ( 1 This is the interpretation of HIPAA, which states that the difference is 0.5 / 2.

[0079] An important consideration when determining the acceptable risk of a system is the intended recipient of the data. Where the receiving pool is more tightly controlled, it may be deemed acceptable to have a higher risk threshold; in contrast, systems that expose data to the public should err towards less risk. In the example of the invention described herein, a conservative risk threshold of 0.05 was deemed appropriate, even though the system does not produce de-identified information for public consumption.

[0080] A key property of the data analytics platform 114 differential privacy system is its ability to track cumulative privacy loss over a series of statistical disclosures. This ability is called composition. Composition allows the system to track the total privacy loss for a particular database. This idea of ​​privacy loss is called a privacy budget, and the tool for tracking it is called privacy accounting and is based on the rigorous mathematics of differential privacy composition techniques. A privacy budget is illustrated in Figure 21.

[0081] The privacy budget is defined by a positive number called epsilon (ε). Each table in a dataset is assigned an epsilon value ("Per-Table Epsilon" in FIG. 20), and each query issued against the dataset must also specify an epsilon value ("Per-Query Epsilon" in FIG. 20). These epsilon values ​​control the amount of noise added to the query's results. The higher the query's ε parameter, the more accurate the results will be but the lower the privacy guarantees, and vice versa. In an exemplary embodiment, a per-table epsilon of 1000 and a per-query epsilon of 1 are selected for the system. As explained below, these choices reduce the privacy risk at these configuration values ​​to below a selected threshold of 0.05.

[0082] When de-identifying data through differential privacy, it is necessary to determine the "scope" of the budget and the conditions under which it can be reset. There are three options for the budget's scope: a "global" budget is a single budget for all data usage; a "project" budget is a separate budget for each project; and a "user" budget is a separate budget for each user or researcher. A key consideration for choosing a budget is determining whether collusion between users is expected to attempt to exfiltrate private data. The system deployment for the illustrated embodiment uses project-level budget tracking because it is unreasonable to expect multiple researchers at an institution with a data usage agreement to collude with each other to launch a privacy attack against the system. Furthermore, the system allows the budget to be reset for a project if researcher activity logs are audited and show no evidence of suspicious activity, indicating benign use of the system, and also when a new version of the data is released.

[0083] According to the privacy budget shown in FIG. 21, the budget starts at ε=100 in this example. The first query has ε=0.1, so the remaining budget is then 99.9. The second query also has ε=0.1, and so on, until the budget is reduced to 0.3. At that point, the researcher runs a query with ε=0.5. Because this value exceeds the remaining privacy budget, the query is blocked by the data analysis platform 114. The data analysis platform 114 includes memory or a hardware register to track this value.

[0084] As explained above, analysts are never directly exposed to protected health information. For this reason, inadvertent patient re-identification (e.g., an analyst recognizing a nearby or former patient) is not a reasonable threat to the system. Therefore, the system is designed to thwart intentional attacks designed to exfiltrate private information as the only legitimate threat vector to the system. In statistical disclosure control, a simple metric for assessing privacy risk from intentional attacks is the probability of a successful privacy attack (Pr(Success)), which is given by the formula: Pr(success)=Pr(success|trial)*Pr(trial) where Pr(attempts) is the probability of an attack and Pr(success|attempts) is the probability that an attack will succeed if an attack is attempted. This metric is adopted to quantify privacy risk in a system.

[0085] The value of Pr(attempts) must be determined via expert opinion, and it is considered best practice to err on the side of conservative assumptions when establishing it. Two aspects to consider when estimating Pr(attempts) are who is attacking (i.e., adversary modeling) and what mitigation controls are in place. The remainder of this section describes the adversary model that presents the most significant risk of attempted privacy attacks, summarizes the mitigation controls in the system, and presents the determined Pr(attempts) given consideration of these factors.

[0086] As previously mentioned, access to the system in the illustrated embodiment is not publicly available; all users of the system are approved through an internal review board and are only granted access to the system for legitimate medical research. (Of course, for other non-medical applications, different safeguards may be employed in alternative embodiments.) A vetting process determined that high-performance attacks against the system from authorized users are not a reasonable threat. That said, to establish conservative privacy risk assumptions for the data analytics platform 114, it was evaluated using high-performance attacks most likely to be performed by privacy researchers. Privacy researchers are the focus because surveys of privacy attacks have found that the majority of attacks are attempted by privacy researchers. The motivation for privacy researchers is not to use the system for its intended research purposes, but to publish compelling research findings related to privacy vulnerabilities.

[0087] To mitigate the likelihood that a researcher will attempt an attack, in the illustrated embodiment, the researcher must partner with an institution that has entered into a data use agreement with the data provider. The agreement imposes an explicit prohibition on re-identification, so a researcher attempting an attack must do so in knowing or misguided violation of the legal agreement. All researcher interactions are logged and periodically audited by the data provider to verify that system use is consistent with the intended research study goals. Privacy attacks have clear and recognizable patterns, such as random number generators used to manipulate SQL statements, highly specific query filters, and fast execution of queries with slight modifications. These types of behavior can be easily spotted by administrators. Finally, researchers are only provisioned with access for the duration of the study, thus mitigating the risk of data being used outside the research context.

[0088] Given the extensive control over the dataset and the fact that an investigator would need to make an error in judgment to attempt to re-identify patient data, the system relies on the estimation that fewer than 1 in 100 investigators provisioned with access to the system (<1%) will attempt a re-identification attack. In accordance with best practice, a conservative correction factor of 10x is applied to the Pr(attempts) value, resulting in a final value of 0.10. Of course, other values ​​may be employed in alternative embodiments of the invention.

[0089] When patient data is de-identified using methods such as randomization or generalization, there is a one-to-one mapping between the de-identified data and the underlying data from which it is derived. This characteristic makes this type of data “person-level.” In the system described herein, there is no explicit one-to-one mapping between the de-identified data and the underlying data from which it is derived. Instead, analysts are only exposed to aggregate data through the data analysis platform 114 and never to person-level data.

[0090] The privacy assessment considered in evaluating the systems and methods described herein uses three measures of privacy: membership disclosure, attribute disclosure, and identity disclosure. Membership disclosure occurs when an attacker can determine that a dataset contains records from a particular patient. Membership disclosure for the present system occurs when a powerful attacker, i.e., one who already possesses the complete records of a set of patients P, can determine whether anyone from P is in the dataset by observing patterns in queries, synthetic data, and / or output from machine learning models. The knowledge gained by an attacker can be limited if the dataset is balanced in its clinical concepts. In other examples, the knowledge gained will be limited if the dataset is balanced in other attributes related to the subjects in the data.

[0091] Attribute disclosure occurs when an attacker can derive additional attributes, such as diagnoses and medications, about a patient based on a subset of attributes already known to the researcher. Attribute disclosure is a more relevant threat because the attacker only needs to know a subset of the patient's attributes.

[0092] Identity disclosure occurs when an attacker is able to link a patient to a specific entry in the database. Direct linkage between a patient and a record allows the attacker to learn all sensitive information contained within the record related to the patient.

[0093] The privacy risk of the system's SQL functionality and the privacy risk of its synthetic data functionality are assessed independently. The privacy risk of machine learning capabilities does not need to be assessed because ML models are never returned to users and therefore do not represent a privacy risk. In theory, it is possible to use information gained from attacking one core functionality to inform attacks on a different core functionality. For example, it seems theoretically plausible to use information learned from synthetic data to inform attacks on an SQL query system. However, there are no known attacks that achieve this goal, and developing one would require a high degree of sophistication, time, and resources. For this reason, such attacks are considered an unreasonable threat to the system and are excluded from establishing Pr(successes|attempts).

[0094] To assess privacy risks for data provider datasets, the system empirically evaluates the privacy risks of datasets in the system. In the remainder of this section, privacy risk assessments for the query engine and synthetic dataset generator are provided. Privacy risks stemming from machine learning models are not assessed because users cannot retrieve or view the model data; they are only evaluation metrics. While the empirical results presented in the following sections use a specific experimental setup, the technical characteristics of the system's privacy mechanisms make the results highly generalizable and therefore an accurate and representative assessment of privacy risks for datasets.

[0095] As mentioned above, the data analytics platform 114 does not allow users to view person-level data. As a result, membership disclosure can only occur by leaking unique attributes about patients in the database. Therefore, the system evaluates attribute disclosure as a primary threat vector for the system's query engine. As mentioned above, this characteristic of the system means that it not only protects against the disclosure of patients' unique attributes, but also maintains the confidentiality of those attributes. To quantitatively establish the risk of attribute disclosure, there are three types of attacks that can be performed against the system: differencing, averaging, and reconstruction. The reconstruction attack is the most sophisticated and powerful attack, and therefore it is used as the basis for establishing a conservative upper bound on the risk posed by queries in the system and compared to the previously identified re-identification risk threshold of 0.05.

[0096] A differential attack aims to single out individuals in a dataset and discover the values ​​of one or more specific attributes. The attack is carried out by running aggregate queries on the target attribute and dataset, both with and without the user. By differencing the results of the user-present and user-absent queries, the attacker attempts to derive the value of the target attribute regardless of the aggregate answer returned by the system.

[0097] Figure 22 shows the results of a differential attack on a system without differential privacy. As can be seen, the attacker is able to derive the correct length of stay for an individual patient using only four lines of code. Figure 23 shows an attempt of the same attack on several embodiments of the present invention. The differential privacy feature prevents this attack from succeeding. The result achieved by the attacker is a length of stay of approximately 512 days (an unreasonably long and clearly incorrect result), while the correct answer is 7.26 days. The most important aspect of configuring a differential privacy mechanism in a system is setting a privacy budget. The chart in Figure 24 shows the results of this attack for varying values ​​of epsilon per query. Although the values ​​obtained vary, all of the values ​​are considered to have resulted in an unsuccessful attack due to their high standard deviation.

[0098] Because the noise addition from the differential privacy mechanism introduces randomness, we can evaluate the results through simulation. The simulation process runs the differential attack 100 times at each epsilon level, allowing the differential privacy mechanism to calculate the noise independently at each iteration. The attack results at each iteration are recorded and plotted for a specific example in Figure 25. The mean and standard deviation of per-query epsilon for all simulations are recorded in the chart in Figure 24. We can see that for per-query epsilons between 1.0 and 10.0, the derived values ​​of the differential attack are useless to an attacker (i.e., the true answer lies outside one standard deviation of the average attack result). At values ​​between 100.0 and 1000.0, the average attack result is much closer to the true answer but still provides the attacker with uncertain and inconclusive results.

[0099] An averaging attack is an attack specifically designed to attack systems that provide noise-protected results. An attacker runs a single query multiple times and simply averages the results. The data analytics platform 114 protects against averaging attacks by "caching" queries, or by ensuring that the exact same noisy results are provided each time the same query is run. The data analytics platform 114 maintains memory and other storage media for this purpose. By providing the exact same results each time a query is run, the data analytics platform 114 does not provide the attacker with a distribution of results to average over. As shown by FIG. 26, only a few lines of code are required to mount an averaging attack to determine the length of stay for a patient. In the caching-free system shown in this figure, an attacker can find that in this simple manner, they can successfully thwart noise added to the system.

[0100] To evaluate the system's robustness against averaging attacks, we simulate the attack in Figure 26 against the database without caching and at varying epsilon levels. The results are shown in the chart in Figure 27. We can observe that queries using epsilon 0.1 are far from the true mean of 20.0. At 1.0, the mean is within about 10% of the true value, but with high variance. At a value of 10.0, the mean closely approximates the true mean, but the standard deviation is about one-tenth of that of the run at 1.0. A key observation in the results is that the number of queries does not substantially affect the accuracy of the attack results. The greatest indicator of successful attack results is query epsilon, not the number of queries used in the attack. Figure 28 shows the results with caching implemented in various embodiments of the present invention. Because caching prevents the use of repeated queries to average over results, i.e., the same results will be returned no matter how many times the same query is run, the chart in Figure 28 only shows results for the first 10 queries.

[0101] Reconstruction attacks can lead to significant privacy violations. They exploit a concept known as the Fundamental Law of Information Recovery, which states that "overly accurate answers to too many questions destroy privacy in spectacular ways." Every time a query is answered by a database, it necessarily releases some information about the data subjects in the database. Reconstruction attacks use linear programming to derive a set of equations that, in their entirety, can reconstruct the attributes (or even the complete database). This process is shown graphically in Figure 29. By running the illustrated queries, an attacker can reconstruct an individual with several attributes to a high degree of accuracy.

[0102] Systems and methods according to some embodiments of the present invention employ a high-performance reconstruction attack, as described in Cohen et al., "Linear Program Reconstruction in Practice," arXiv:1810.05692v2 [cs.CR], January 23, 2019, which is incorporated herein by reference. The attack attempts to completely reconstruct the values ​​of clinical attribute columns for a given range of patient identifiers based on the results of a series of aggregate queries.

[0103] Note that researchers accessing the data analysis platform 114 do not have permission to implement filters on identifiers such as patient identifiers. However, a motivated attacker could attempt to single out patients using other means, such as using overly specific filter conditions in the researcher's query. Doing so would be suboptimal for approximating the range of patient identifiers. By employing patient identifiers in reconstruction attack experiments, the system establishes a worst-case scenario estimate of privacy leakage in the system.

[0104] The attack focused on a selected range of 100 patient identifiers, with each query counting binarized clinical attribute values ​​across at least 35 pseudo-randomly selected patients within the identifier range. The baseline effectiveness of the attack was measured by running queries against a database that did not provide privacy protection. With 1000 queries, the attack against the unprotected database was able to reconstruct the binarized clinical attributes for the patients with perfect accuracy.

[0105] The same attack was then performed on the clinical dataset in our system using three different levels of total differential privacy epsilon budget: 100, 1,000, and 10,000. The entire budget was allocated to the same 1,000 queries that enabled perfect reconstruction in the unprotected database (1 / 1,000 of the budget for each query). The precision, recall, and accuracy of clinical attribute reconstruction were evaluated using 20 trials at each budget level, and the resulting distributions are shown in the graphs in Figure 30. As can be seen, at a per-query epsilon of 10.0, the attack is able to reconstruct the binarized clinical attributes for 100 patients with near-perfect accuracy. At per-query epsilon values ​​of 0.1 and 1.0, the attacker is unable to derive conclusive results regarding the attributes. Thus, the experiments demonstrate that the differential privacy epsilon budget provides an effective means of mitigating reconstruction attacks at per-query epsilon values ​​of 0.1 and 1.0.

[0106] Figure 31 provides a distribution chart showing the data from Figure 30 in a different format, with patient stays spread across the x-axis and the darkness of the vertical bar indicating how often each stay was predicted to have a positive value for the clinical attribute, with darker colors indicating more frequent predictions. The true values ​​(ground truth) of the clinical attributes are shown in the bottom row. The lighter coloring in the top row indicates the attacker's uncertainty in their level of epsilon per query. As can be seen, the epsilon value significantly impacts the attacker's ability to succeed.

[0107] While the above analysis provides significant confirmation of the effectiveness of the data analytics platform 114 in foiling attacks, it remains to relate the query re-identification risk to regulatory thresholds, such as applicable HIPAA thresholds. The methodology described herein measures the probability that a reconstruction attack will be successfully executed against a system. As previously mentioned, the reconstruction attack was chosen as the attack model for establishing the estimated re-identification risk because it represents the most sophisticated attack against a system to attempt to exfiltrate private information.

[0108] The probability of a successful attack is measured by simulating many reconstruction attacks and determining what percentage of those attacks are successful. For this purpose, the term "successful" attack is defined as one that is capable of outperforming the baseline classifier. The chart in Figure 32 records the results of these simulations. As shown, attacks against systems configured with epsilon per query of 0.1 and 1.0 are unsuccessful 100% of the time; therefore, Pr(Success) is below the chosen threshold of 0.05. At epsilon per query of 10.0, attacks are successful 72% of the time; therefore, Pr(Success) exceeds the target threshold, which is not the chosen configuration for the system.

[0109] Two methods are employed to assess the privacy risk of synthetic data capabilities in the system. The first is an attribute inference attack, and the second is an identity disclosure risk measure. Identity disclosure represents the most significant privacy violation in the system, and it is used as the basis for the Pr(success) metric for synthetic data capabilities.

[0110] In the case of attribute inference attacks, it is assumed that an attacker somehow gains access to some or all of the original data, but that such an investigator only somehow obtains a subset of the features and wishes to infer the missing information, similar to the setup with the query reformulation attack described above. Because the attacker also has access to synthetic data containing all the features, the attacker can attempt to infer missing values ​​in the original data by using similar records in the synthetic data. This seems plausible, since the synthetic data is expected to exhibit the same statistical properties as the original data. The results of the attack are shown in the chart in Figure 33. As can be seen, the attack is largely unsuccessful, regardless of whether the k-nn or random forest method is chosen. Regularization (i.e., dropout) reduces the attack performance.

[0111] To establish the identity disclosure risk posed by a synthetic dataset, a risk metric is employed. The risk metric works by rigorously comparing the generated synthetic dataset with the original dataset from which it was derived, producing a conservative estimate of the identity disclosure risk posed by the synthetic dataset. The metric considers several factors related to the dataset, including the number of records in the derived synthetic dataset that match records in the original dataset, the probability of error in the original dataset, and the probability that an attacker can verify that the matched records are accurate. As shown in Figure 34, the system is consistently able to produce synthetic datasets with identity disclosure risk far below the target of 0.05. Synthetic data is generated from clinical datasets at both a very low regularization level (dropout = 0.0) and a high regularization level (0.5). Both are an order of magnitude lower than the target for the upper bound on re-identification risk, with the probability of a successful attack (i.e., Pr(Success)) being less than 10 percent in both cases.

[0112] The expert determination described herein relies on empirical results to quantify the privacy risk associated with the use of the system. It is not possible to conduct every conceivable experiment on the target dataset before releasing it to users. Therefore, the generalizability of the observed empirical results must be considered. The empirical results are a strong representation of the overall privacy risk in the system for two reasons.

[0113] The first is the concept of sensitivity in differential privacy. This property of the technique adjusts the noise added to statistical results based on the re-identification risk presented by each query. This means that the system accounts for the total privacy loss, rather than the total noise added. Thus, the privacy risk remains roughly constant across different datasets and queries.

[0114] Second, the empirical evaluations described in some embodiments of the invention described herein are considered conservative, i.e., they employ attacks that are much more sophisticated than what could reasonably be expected to be launched against the system. Furthermore, the system in some embodiments employs a conservative risk threshold of 0.05, which is about one-tenth the size of other systems used to de-identify health data. For these reasons, it is considered unreasonable to expect that the observed re-identification risk for different (but similar) clinical datasets would significantly exceed the reported re-identification risk values ​​described herein.

[0115] Another issue is dataset growth over time. A typical clinical dataset grows continuously as patient encounters are retained in its data warehouse. Each additional patient encounter can contribute meaningful information to researchers. New patient encounters can be added to the dataset on a "batch" basis, with the target of adding a new batch of patient data every one to three months. Each of these incremental datasets becomes a new "version" of the clinical dataset. It is necessary to evaluate the degree to which the re-identification risk of one version represents the re-identification risk of successive versions. This should be considered in the context of both queries and synthetic data. The generalization property of a differential privacy system, as explained above, means that queries in the system are expected to create approximately the same re-identification risk across each version of the dataset. For synthetic datasets, because the re-identification risk is dynamically measured for each dataset, the system dynamically enforces that the re-identification risk is within the established target for the generated report.

[0116] Note that there are multiple points of information disclosure described in the above workflow. These disclosures can include the release of information from the dataset to external investigators as well as the publication of findings. The privacy guarantees of differential privacy are sensitive to information disclosure, which means that as more information about the protected dataset is disclosed, even just aggregate, non-identifiable data, the privacy guarantees provided by differential privacy are weakened. If enough information is released, an attacker can use that information in constructing an attack against the database.

[0117] For example, an external researcher can ask the following queries and get the following answers: Q1=COUNT(X AND Y) R1=16 Q2=COUNT(X AND Y AND Z) R2=14

[0118] Both R1 and R2 are differentially private, which implies that COUNT(X AND Y AND NOT Z) is 2. Now imagine that a paper publishes the true counts of X and Y. T1=COUNT(X AND Y) P1=12

[0119] Because COUNT(X AND Y) and COUNT(X AND Y AND Z) are correlated, the outside investigator clearly learns that R2 cannot possibly be 14. What the outside investigator now knows is that the true value for R2 is somewhere in the interval (0,12). In this example, if X=(gender=male), Y=(sex=woman), and Z=(age=22), then publishing the non-differential private result has contributed to an information gain for the attacker without requiring a widespread and sophisticated attack.

[0120] While it can be acknowledged that periodic publication of non-private aggregate statistics about a dataset could potentially weaken the overall privacy guarantees the system provides to external researchers, these types of privacy attacks are not a reasonable assumption for two reasons. The first is the complexity of multiple versions of the data being created throughout the project lifecycle, combined with the fact that external researchers only have access to the system during the course of their research. As a result, it is unlikely that external researchers would have access to their exact version of the data at the same time that published non-private results are available to such researchers to launch a privacy attack. Second, in some embodiments, users of the system are generally evaluated and vetted by research experts and are subject to data use agreements that prohibit attempts at re-identification.

[0121] The systems and methods described herein may, in various embodiments, be implemented by any combination of hardware and software. For example, in one embodiment, the systems and methods may be implemented by a computer system or collection of computer systems, each of which includes one or more processors that execute program instructions stored on a computer-readable storage medium coupled to the processors. The program instructions may implement the functions described herein. The various systems and displays described herein, as illustrated in the figures, represent example implementations. The order of any method may be changed, and various elements may be added, modified, or omitted.

[0122] The computing systems or computing devices described herein may implement the hardware portion of a cloud computing system or a non-cloud computing system, forming part of various implementations of the present invention. The computing system may be any of a variety of types of devices, including, but not limited to, a commodity server, a personal computer system, a desktop computer, a laptop or notebook computer, a mainframe computer system, a handheld computer, a workstation, a network computer, a consumer device, an application server, a storage device, a telephone, a mobile phone, or generally any type of computing node, compute node, compute device, and / or computing device. The computing system includes one or more processors (which may include multiple processing cores, any of which may be single- or multi-threaded) coupled to system memory via an input / output (I / O) interface. The computing system may further include a network interface coupled to the I / O interface.

[0123] In various embodiments, the computer system may be a single-processor system including one processor or a multiprocessor system including multiple processors. The processor may be any suitable processor capable of executing computing instructions. For example, in various embodiments, the processor may be a general-purpose processor or an embedded processor implementing any of a variety of instruction set architectures. In a multiprocessor system, each of the processors may typically, but not necessarily, implement the same instruction set. The computer system also includes one or more network communication devices (e.g., network interfaces) for communicating with other systems and / or components across a communications network, such as a local area network, a wide area network, or the Internet. For example, a client application executing on a computing device may use the network interface to communicate with a server application executing on a single server or on a cluster of servers implementing one or more of the components of the systems described herein in various subsystems, in a cloud computing or non-cloud computing environment. In another example, an instance of a server application executing on a computer system may use the network interface to communicate with other instances of the application, which may be implemented on other computer systems.

[0124] A computing device also includes one or more persistent storage devices and / or one or more I / O devices. In various embodiments, the persistent storage device may correspond to a disk drive, a tape drive, solid-state memory, other mass storage device, or any other persistent storage device. A computer system (or a distributed application or operating system running thereon) may store instructions and / or data in the persistent storage device as desired and retrieve the stored instructions and / or data as needed. For example, in some embodiments, a computer system may implement one or more nodes of a control plane or control system, and the persistent storage may include SSDs attached to the server nodes. Multiple computer systems may share the same persistent storage device or may share a pool of persistent storage devices, where the devices in the pool represent the same or different storage technologies.

[0125] A computer system includes one or more system memories that can store code / instructions and data accessible by the processor(s). The system's memory capabilities may include, for example, multiple levels of memory and memory caches in systems designed to swap information in memory based on access speed. Interleaving and swapping may extend to persistent storage in virtual memory implementations. Technologies used to implement memory may include, by way of example, static random-access memory (RAM), dynamic RAM, read-only memory (ROM), nonvolatile memory, or flash-type memory. As with persistent storage, multiple computer systems may share the same system memory or may share a pool of system memory. One or more system memories may contain program instructions executable by the processor(s) to implement the routines described herein. In various embodiments, the program instructions may be encoded in binary, assembly language, any interpreted language such as Java, a compiled language such as C / C++, or any combination thereof, and the particular languages ​​given herein are merely illustrative. In some embodiments, the program instructions may implement multiple separate client, server nodes, and / or other components.

[0126] In some implementations, the program instructions may include executable instructions to implement an operating system (not shown), which may be any of various operating systems, such as UNIX, LINUX, Solaris, MacOS, or Microsoft Windows. Any or all of the program instructions may be provided as a computer program product or software, which may include a non-transitory computer-readable storage medium having stored thereon instructions that may be used to program a computer system (or other electronic device) to perform a process according to various implementations. A non-transitory computer-readable storage medium may include any mechanism for storing information in a form (e.g., software, processing application) readable by a machine (e.g., a computer). Generally, a non-transitory computer-accessible medium may include a computer-readable storage medium or memory medium, such as a magnetic medium or an optical medium, e.g., a disk or DVD / CD-ROM coupled to a computer system via an I / O interface. Non-transitory computer-readable storage media may include any volatile or non-volatile media, such as RAM or ROM, that may be included as system memory or another type of memory in some embodiments of a computer system. In other implementations, program instructions may be communicated using optical, acoustic, or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.) conveyed over a communications medium, such as a network and / or wired or wireless link, such as may be implemented via a network interface. The network interface may be used to interface with other devices, which may include other computer systems or any type of external electronic device.Generally, system memory, persistent storage, and / or remote storage accessible on other devices through a network may store data blocks, replicas of data blocks, metadata associated with the data blocks and / or their state, database configuration information, and / or any other information usable in implementing the routines described herein.

[0127] In some implementations, the I / O interface may coordinate I / O traffic between the processor, system memory, and any peripheral devices in the system, including through a network interface or other peripheral interface. In some embodiments, the I / O interface may perform any necessary protocol, timing, or other data conversions to convert data signals from one component (e.g., system memory) into a format suitable for use by another component (e.g., processor). In some embodiments, the I / O interface may include support for devices attached through various types of peripheral buses, such as variations on the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard. Also, in some embodiments, some or all of the functionality of the I / O interface, such as the interface to system memory, may be incorporated directly into the processor(s).

[0128] The network interface may be, for example, a network interface through which data may be transmitted between a computer system and one or more storage system server nodes, primary nodes, read-only nodes, etc., as described herein. Do,The I / O interface may enable data to be exchanged with other network-attached devices, such as other computer systems (which may implement a client of a database system and / or a database system). Additionally, the I / O interface may enable communication between the computer system and various I / O devices and / or remote storage. The I / O devices, in some embodiments, may include one or more display terminals, keyboards, keypads, touchpads, scanning devices, voice or light recognition devices, or any other devices suitable for entering or retrieving data with one or more computer systems. These may be directly connected to a particular computer system or may generally be connected to multiple computer systems in a cloud computing environment, a grid computing environment, or other system involving multiple computer systems. Multiple I / O devices may be in communication with a computer system or may be distributed on various nodes of a distributed system that includes the computer system. The user interfaces described herein may be visible to a user using various types of display screens, which may include CRT displays, LCD displays, LED displays, and other display technologies. In some implementations, input may be received through a display using touchscreen technology, while in other implementations, input may be received through a keyboard, mouse, touchpad, or other input technology, or any combination of these technologies.

[0129] In some embodiments, similar input / output devices may be separate from the computer system and may interact with one or more nodes of a distributed system that includes the computer system through a wired or wireless connection, such as over a network interface. The network interface may typically support one or more wireless networking protocols (e.g., Wi-Fi / IEEE 802.11 or another wireless networking standard). The network interface may support communication over any suitable wired or wireless general data network, such as, for example, other types of Ethernet networks. Furthermore, the network interface may support communication over a telecommunications / telephony network, such as an analog voice network or a digital fiber communications network, a storage area network, such as a Fibre Channel SAN, or any other suitable type of network and / or protocol.

[0130] Any of the distributed system embodiments described herein, or any of their components, may be implemented as one or more network-based services in a cloud computing environment. For example, read-write and / or read-only nodes in the database tier of a database system may present database services and / or other types of data storage services employing the distributed storage system described herein to clients as network-based services. In some embodiments, a network-based service may be implemented by software and / or hardware systems designed to support interoperable machine-to-machine interaction over a network. A web service may have an interface described in a machine-processable format, such as the Web Services Description Language (WSDL). Other systems may interact with the network-based service in a manner prescribed by the network-based service's interface description. For example, a network-based service may define various operations that other systems can invoke and may define specific application programming interfaces (APIs) that other systems may be expected to follow when requesting the various operations.

[0131] In various embodiments, network-based services may be requested or invoked through the use of messages containing parameters and / or data associated with a network-based service request. Such messages may be formatted according to a particular markup language, such as Extensible Markup Language (XML), and / or may be encapsulated using a protocol, such as Simple Object Access Protocol (SOAP). To perform a network-based service request, a network-based service client may assemble a message containing the request and communicate the message to an addressable endpoint (e.g., a Uniform Resource Locator (URL)) corresponding to the web service using an Internet-based application layer transport protocol, such as Hypertext Transfer Protocol (HTTP). In some embodiments, network-based services may be implemented using Representational State Transfer (REST) ​​techniques rather than message-based techniques. For example, a network-based service implemented according to REST techniques may be invoked through parameters contained within an HTTP method, such as PUT, GET, or DELETE.

[0132] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Although methods and materials similar or equivalent to those described herein can also be used in the practice or testing of the present invention, a limited number of exemplary methods and materials are described herein. It will be apparent to those skilled in the art that many more modifications are possible without departing from the inventive concepts herein.

[0133] All terms used herein should be interpreted in the broadest possible manner consistent with the context. When groupings are used herein, all individual members of the group, and all possible combinations and subcombinations of the group, are intended to be individually included. When ranges are specified herein, the ranges are intended to include all subranges and individual points within the range. All references cited herein are incorporated by reference to the extent that they are not inconsistent with the present disclosure herein.

[0134] The present invention has been described in terms of several preferred and alternative embodiments, which are intended to be examples only and not limitations on the full scope of the invention as set forth in the appended claims.

Claims

1. 1. A system for supporting hypothesis testing on private data, the system comprising: at least one processor; a database in communication with the at least one processor, the database comprising a private data set; at least one non-transitory medium in communication with the processor; the non-transitory medium comprises an instruction set comprising instructions that, when executed on the at least one processor in communication with the at least one processor, receiving a request for an operation on the private data set from an external analysis computer system; applying differential privacy to compute an answer to the request from the external analysis computer system; applying noise to the answers from the external analysis computer system to produce noisy results using differential privacy; and returning the noisy results to the external analysis computer system; receiving a request from the external analysis computer system for hypothesis testing against the private data set; performing a hypothesis test on the private data set; computing a final result from the performance of the hypothesis test on the private dataset, the final result preserving the privacy of the private dataset through the application of differential privacy, the final result being a set of aggregate statistical results; returning said final results to said external analysis computer system; A system configured to:

2. 10. The system of claim 1, wherein the set of instructions, when executed on the at least one processor in communication with the at least one processor, is further configured to: provide the final results to an internal analysis computer system; and, if approval is received from the internal analysis computer system, release the final results to the external analysis computer system.

3. When the set of instructions is executed on the at least one processor in communication with the at least one processor, receiving a query from the internal analysis computer system; executing the query against the private dataset; returning a set of true results to said internal analysis computer system; The system of claim 2 , further configured to:

4. When the set of instructions is executed on the at least one processor in communication with the at least one processor, receiving a machine learning training or evaluation request from the external analysis computer system; capturing the machine learning training or evaluation request and performing a machine learning task on the private dataset; returning machine learning training or evaluation results to said external analysis computer system; The system of claim 1 , further configured to:

5. The system of claim 4 , wherein the machine learning training or evaluation results include summary statistics without a machine learning model.

6. When the set of instructions is executed on the at least one processor in communication with the at least one processor, receiving a machine learning training or evaluation request from an internal analytics computer system; capturing the machine learning training or evaluation request and performing a machine learning task on the private dataset; returning a set of summary statistics to said internal analysis computer system; receiving a request for a machine learning model from the internal analysis computer system; Retrieving and returning the machine learning model to the internal analysis computer system; The system of claim 2 , further configured to:

7. The system of claim 6 , wherein the machine learning training or evaluation results include summary statistics without a machine learning model.

8. When the set of instructions is executed on the at least one processor in communication with the at least one processor, receiving a request for a synthetic data set from the external analysis computer system; generating a synthetic dataset from the private dataset; assessing the privacy of the synthetic dataset; and returning the synthetic dataset to the external analysis computer system based on the results of the evaluation of the privacy of the synthetic dataset; The system of claim 1 , further configured to:

9. 2. The system of claim 1, wherein the instruction set, when executed on the at least one processor in communication with the at least one processor, is further configured to: calculate a query epsilon budget for each request for an operation on the private dataset from the external analysis computer system; apply a per-query epsilon budget; and return the noisy results to the external analysis computer system only if the per-query epsilon budget is not exceeded by the request for an operation.

10. 10. The system of claim 9, wherein the instruction set, when executed on the at least one processor in communication with the at least one processor, is further configured to: maintain a per-project epsilon budget; calculate a query epsilon for multiple requests for operations on the private dataset from the external analysis computer; calculate a project epsilon by adding each previous query epsilon; and return the noisy result to the external analysis computer only if the project epsilon does not exceed the per-project epsilon budget.

11. 11. The system of claim 10, wherein the instruction set, when executed on the at least one processor in communication with the at least one processor, is further configured to calculate at least one of a per-query epsilon budget and a project epsilon budget by simulating a privacy attack on the private dataset.

12. The system of claim 11 , wherein the privacy attack is a linear programming reconstruction attack.

13. The system of claim 12 , wherein the epsilon is set to correspond to a probability of success for the privacy attack of 0.05 or less.

14. 1. A method for testing hypotheses using private data, the method comprising: receiving, at a data analysis platform, a request from an external analysis computer system for an operation on a private dataset stored in a database connected to the data analysis platform; applying differential privacy in the data analytics platform to compute an answer to the request from the external analytics computer system; applying noise to the answers from the external analysis computer system to produce noisy results in the data analysis platform; returning the noisy results from the data analysis platform to the external analysis computer system; receiving, at the data analysis platform, a request from the external analysis computer system to test a hypothesis on the private data set; performing a hypothesis test on the private data set in the database in the data analysis platform; calculating, in the data analysis platform, a final result from the performance of the hypothesis test on the private dataset, the final result preserving the privacy of the private dataset through the application of differential privacy, the final result being a set of aggregate statistical results; returning the final results from the data analysis platform to the external analysis computer system; A method comprising:

15. providing the final results from the data analysis platform to an internal analysis computer system; releasing the final results from the data analysis platform to the external analysis computer system if approval is received from the internal analysis computer system; 15. The method of claim 14, further comprising:

16. receiving, at the data analysis platform, a query from the internal analysis computer system; executing the query on the private dataset in the data analysis platform; returning a set of true results from the data analysis platform to the internal analysis computer system; 16. The method of claim 15, further comprising:

17. receiving, at the data analytics platform, a machine learning training or evaluation request from the external analytics computer system; capturing the machine learning training or evaluation request in the data analytics platform; performing machine learning tasks on the private data set in the database in the data analytics platform; returning machine learning training or evaluation results from the data analysis platform to the external analysis computer system; 15. The method of claim 14, further comprising:

18. 20. The method of claim 17, wherein the machine learning training or evaluation results include summary statistics without a machine learning model.

19. receiving, at the data analytics platform, a machine learning training or evaluation request from an internal analytics computer system; capturing the machine learning training or evaluation request in the data analytics platform; performing machine learning tasks on the private dataset in the data analytics platform; returning a set of summary statistics from the data analysis platform to the internal analysis computer system; receiving, at the data analytics platform, a request for a machine learning model from the internal analytics computer system; retrieving, in the data analytics platform, the machine learning model and returning it to the internal analytics computer system; 16. The method of claim 15, further comprising:

20. 20. The method of claim 19, wherein the machine learning training or evaluation results include summary statistics without a machine learning model.

21. receiving, at the data analysis platform, a request for a synthetic dataset from the external analysis computer system; generating a synthetic dataset from the private dataset in the database in the data analysis platform; assessing the privacy of the synthetic dataset in the data analytics platform; sending the synthetic dataset from the data analysis platform to the external analysis computer system based on the result of the assessment of the privacy of the synthetic dataset; 15. The method of claim 14, further comprising:

22. 15. The method of claim 14, further comprising: calculating a query epsilon budget for each request for an operation on the private dataset from the external analysis computer system to the data analysis platform; applying a per-query epsilon budget; and returning the noisy results to the external analysis computer system only if the per-query epsilon budget is not exceeded by the request for an operation.

23. maintaining a per-project epsilon budget in the data analytics platform; calculating, in the data analysis platform, query epsilons for a plurality of requests for operations on the private dataset from the external analysis computer; calculating a project epsilon in the data analysis platform by adding each of the previous query epsilons and returning the noisy result to the external analysis computer only if the project epsilon does not exceed the per-project epsilon budget; 23. The method of claim 22, further comprising:

24. 24. The method of claim 23, further comprising: calculating, in the data analytics platform, at least one of a per-query epsilon budget and a project epsilon budget by simulating a privacy attack on the private dataset.

25. 25. The method of claim 24, wherein the privacy attack is a linear programming reconstruction attack.

26. 26. The method of claim 25, wherein the epsilon is set to correspond to a probability of success for the privacy attack of 0.05 or less.

Images