Protecting input datasets from being linked to further datasets

Abstract

The present disclosure relates to protecting an input dataset from linking with additional datasets. A processor of a computer system calculates multiple values ​​of one or more parameters of a perturbation function, the perturbation function configured to perturb the input dataset to protect the input dataset from linking with additional datasets, each of the multiple values ​​of the one or more parameters of the perturbation function indicating a level of protection against linking with additional datasets. The processor then generates multiple derived datasets from the input dataset and calculates a utility score for each of the multiple derived datasets indicating the utility of the derived dataset for a desired data analysis. The processor then outputs one of the multiple derived datasets with the highest utility score.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to Australian Provisional Patent Application No. 2022 / 902837, filed 30 September 2022, the contents of which are incorporated herein by reference in their entirety.

[0002] The present disclosure relates to protecting an input dataset from being linked to further datasets. [Background technology]

[0003] An increasing amount of data is collected by various different entities, but that data is often not optimally utilized because it remains within the collecting entity. It would be advantageous if data from different entities could be combined. However, what prevents sharing datasets is that it is often possible to link datasets, and that information could be obtained even if it was held securely by each entity and not shared. In other words, linking datasets could lead to the discovery of data that was intended to be held securely from access by unauthorized third parties.

[0004] For example, government agencies have an obligation to share data collected in the public interest. On the other hand, government agencies have personal data that must be kept secure. It is difficult for government agencies and other data collection entities to share some data while ensuring that unshared data remains protected. In particular, it is difficult to protect shared data from linking with other datasets that would reveal shared data, such as re-identification.

[0005] For example, a tax office may have an income database that includes fields for an individual's name, zip code, occupation, and income. The tax office may decide to remove the name field and release the remaining data—occupation, zip code, and income—as "de-identifying data." However, there may only be one surgeon in a particular zip code, and another physician dataset may contain the name of the surgeon for that particular zip code. Thus, the two datasets can be linked—that is, one or more fields whose values ​​exactly match—in this example, the zip code. The result is the name of the surgeon from the physician dataset uniquely linked with the income from the tax dataset. Thus, this linking reveals the exact income of a particular individual, but that information is being withheld by the tax office. Determining how to share a dataset while protecting it from linking with other datasets can be challenging.

[0006] Any discussion of documents, acts, materials, devices, articles or the like which has been included in this specification should not be deemed an admission that any or all of such matters form part of the prior art base or are common general knowledge in the art relevant to the present disclosure as they existed before the priority date of each claim of this application. Summary of the Invention

[0007] The present disclosure provides a system and method for protecting an input dataset from linking with additional datasets. This is achieved by perturbing the input dataset multiple times with multiple different perturbation parameters to generate multiple perturbed datasets, each satisfying a given protection against linking. The disclosed system and method then selects the perturbed dataset with the highest utility for a particular purpose. This approach yields the most useful dataset under a given protection against linking, while also improving computational efficiency because the number of randomizations is reduced. More specifically, highly randomizing a dataset means that a large amount of computing power is used to perturb the dataset. However, in the disclosed solution, the dataset is randomized to a lower degree, significantly reducing the amount of computing resources required.

[0008] A computer-implemented method for protecting an input dataset from linking with further datasets comprises: calculating a plurality of values ​​of one or more parameters of a perturbation function, the perturbation function configured to perturb the input dataset to protect the input dataset from linking with further datasets, each of the plurality of values ​​of the one or more parameters of the perturbation function indicating a level of protection against linking with further datasets; generating a plurality of derived datasets from the input dataset, each of the plurality of derived data sets is generated by applying a perturbation function to the input data set; generating a plurality of derived data sets, each of which is generated by using a different one of a plurality of values ​​of one or more parameters of the perturbation function; calculating, for each of the plurality of derived data sets, a utility score indicative of the utility of the derived data set for a desired data analysis; and outputting one of the plurality of derived data sets having the highest usefulness score.

[0009] In some embodiments, the method further includes receiving a request for the data set from a requestor, wherein the level of protection is based on one or more of the requestor or the data in the request.

[0010] In some embodiments, calculating the multiple values ​​of one or more parameters of the perturbation function is based on a connectivity indicator (PIF) of the input data sets.

[0011] In some embodiments, the method further includes calculating a plurality of cell surprise factors (CSFs), each CSF representing the indistinguishability of an attribute in the input dataset, and calculating a factor indicative of the degree of linkage of the input dataset by combining the plurality of CSFs.

[0012] In some embodiments, the method further comprises partitioning the input dataset into a first partition of quasi-identifiers and a second partition of sensitive data, and the perturbation function is applied only to the second partition.

[0013] In some embodiments, the method further includes calculating a coefficient indicating a degree of connection of a second partition that includes one attribute from the first partition, and selectively adding one attribute from the first partition to the second partition based on the calculated coefficient, wherein the perturbation function is applied only to the second partition that includes the selectively added attribute from the first partition.

[0014] In some embodiments, the method further comprises performing fuzzy interference using coefficients indicative of the degree of connectivity of the input data sets to determine multiple values ​​of the one or more parameters.

[0015] In some embodiments, implementing fuzzy interference is based on a coefficient indicating the degree of linkage and a fuzzy membership function for each of the one or more parameters of the perturbation function.

[0016] In some embodiments, the degree of linkage is measured in terms of differential ε, δ privacy, where one or more parameters of the perturbation function are ε and δ.

[0017] In some embodiments, the method further comprises removing the identifier attribute from the input data set.

[0018] In some embodiments, calculating the utility score includes calculating a distribution difference between the input dataset and the derived dataset, and outputting one of the derived datasets having the highest distribution difference.

[0019] In some embodiments, calculating the utility score includes calculating the accuracy of the desired data analysis for the derived datasets and outputting one of the multiple derived datasets having the highest accuracy.

[0020] In some embodiments, the method further includes applying a threat model to the derived dataset with the highest utility score to evaluate similarities between tuples of the input dataset and the derived dataset, and selectively blocking output based on evaluating the similarities.

[0021] In some embodiments, calculating the utility score is based on utility loss and privacy leakage.

[0022] In some embodiments, the utility score is a weighted sum of the utility loss and the privacy leakage.

[0023] In some embodiments, the method selectively blocks output of the derived data set upon determining that the weighted sum of the utility loss and privacy leakage is above a predetermined threshold.

[0024] The software, when executed by a computer, causes the computer to perform the above-described methods.

[0025] A computer system comprising a processor is programmed to carry out the above method.

[0026] Throughout this specification the word "comprise" or variations such as "comprises" or "comprising" will be understood to imply the inclusion of a stated element, integer, or step, or group of elements, integers, or steps, but not the exclusion of any other element, integer, or step, or group of elements, integers, or steps.

[0027] Examples are described with reference to the following drawings: [Brief explanation of the drawings]

[0028] [Figure 1] 1 illustrates a flowchart of an aspect of the present disclosure, according to one embodiment. [Figure 2a] 1 illustrates a method for protecting an input dataset from linking with additional datasets, according to one embodiment. [Figure 2b] 1 illustrates a computer system for protecting an input dataset from linking with additional datasets, according to one embodiment. [Figure 3] 1 illustrates a mapping between three fuzzy variables and the change in personal information factor (PIF) for changes in δ and ε, according to one embodiment. [Figure 4a] 1 illustrates an example algorithm for generating a privacy-preserving dataset, according to one embodiment. [Figure 4b] 1 illustrates an example algorithm for generating a privacy-preserving dataset, according to one embodiment. [Figure 5]

[0023] Figure 5 illustrates a Cell Surprise Factor (CSF) and PIF analysis of an input dataset, and a CSF and PIF analysis of a Q attribute, according to one embodiment. More specifically, Figure 5a illustrates a CSF analysis for the input dataset. Figure 5b illustrates a PIF analysis for the input dataset. The light bars (1st, 7th, 10th, 12th) represent the Q attribute. Figure 5c illustrates a CSF analysis for the Q attribute. Figure 5d illustrates a PIF analysis for the Q attribute. [Figure 6] CSF and PIF analysis of a refined set of Q attributes according to one embodiment. Figure 6a illustrates a CSF analysis for a refined set of Q attributes. Figure 6b illustrates a PIF analysis for a refined set of Q attributes. [Figure 7]

[0033] Figure 7a illustrates the results of a usability analysis for a privacy-preserving dataset. Figure 7b illustrates the results of an effectiveness analysis for a privacy-preserving dataset. [Figure 8] Comparison between utility and effectiveness of privacy-preserving datasets without Q-attribute refinement, according to one embodiment. Figure 8a illustrates the results of a utility analysis for privacy-preserving datasets without Q-attribute refinement. Figure 8b illustrates the results of an effectiveness analysis for privacy-preserving datasets without Q-attribute refinement. [Figure 9] 1 illustrates a server implementation according to one embodiment. [Figure 10] 1 illustrates exemplary configuration data for implementing the disclosed method, according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0029] Sharing data linked with personally identifiable information (PII) can lead to the leakage of sensitive personal information by linking with additional datasets to re-identify datasets that were anonymized before sharing, thus introducing a potential threat to user privacy. Throughout this disclosure, linking or linking refers to the use of any external data to infer information about individual rows. For example, re-identification using external data is an example of linking a dataset with an additional dataset.

[0030] Differential privacy is an exemplary disclosure control mechanism due to its strict privacy guarantees. An algorithm M satisfies differential privacy if, for all adjacent datasets x and y and all possible outputs S, Pr[M(x)∈S]≦exp(ε)Pr[M(y)∈S]+δ, where ε is called the privacy budget and denotes privacy leakage, while δ represents the probability of model failure.

[0031] In a similar notation, we can say that for a mechanism to satisfy (ε,δ)-differential privacy, it satisfies the following equation, where d and d' are datasets that differ by one record:

number

number

[0032] In some examples, tabular data sharing is considered because tabular data is often shared between different institutions or made public for public use or interaction with a specific institution. This disclosure focuses on privacy and utility of tabular data sharing with DP (also referred to as non-interactive data sharing), where the privacy level is quantified using DP and utility is quantified using U(D), which is the application-specific utility (e.g., accuracy, precision) of application A when it is run on D. In a tabular dataset, each row represents an individual (data owner), and the columns represent the features considered under the corresponding set of data owners in the table. Furthermore, in some examples, all rows may be assumed to be independent (belonging to only one owner) and not linked to any other rows (e.g., trajectory data).

[0033] Non-interactive data sharing presents significant challenges due to the extreme level of randomization required to maintain sufficient privacy (acceptable ε values) during data sharing, resulting in less utility being generated from the shared private data (e.g., perturbed tabular data) and excessive required computing resources. Despite its complexity and difficulty, non-interactive data sharing is useful for enabling a wide variety of opportunities from the entire dataset available for analyst analysis, so that the application at hand (e.g., classification, regression, descriptive statistics) is not constrained to a single output (e.g., mean).

[0034] Selecting the best DP approach for differentially private non-interactive data sharing faces several challenges. Some of these challenges include the diversity of input datasets (e.g., statistical properties, dimensionality), the diversity of different types of applications at hand (e.g., data clustering, deep learning), and the potential for unexpected privacy leakage due to the full dataset being made public. Furthermore, there is no framework-based solution that allows DP approaches to be evaluated for their performance for non-interactive data sharing with high utility and high privacy under strict privacy guarantees.

[0035] In some cases, there are unexpected data leaks due to the relaxation of privacy constraints (ε and δ) when achieving high utility. Furthermore, DP non-interactive data sharing, where part of the dataset (a carefully selected set of attributes) is made public for mandatory reasons, has not been investigated so far. This issue may be important in real-world scenarios, such as those used in cross-agency data sharing settings. The availability of unperturbed vertical partitions in the final dataset would provide improved utility for applications based on custom queries and reduce the required computational resources.

[0036] However, this type of setting requires deeper and more rigorous analysis from the perspectives of privacy and attack resilience. This problem is referred to as controlled partially perturbed non-interactive data sharing (CPNDS). Therefore, a framework that facilitates CPNDS in an application-specific, usability- and privacy-preserving manner is desirable. Challenges in CPNDS include (1) availability of a range of complex dynamics in input data (e.g., categorical / non-categorical attributes, IID data, non-IID), (2) maintaining the usability of output datasets for different types of applications required by analysts, and (3) maintaining a balance between privacy and usability (enabling high usability while maintaining a high level of privacy).

[0037] The present disclosure provides a unified, multi-criteria based solution for identifying the best perturbed instance of an input dataset under CPNDS. In some embodiments, the disclosed method is performed under a central authority (e.g., government agency, hospital, bank) that has full ownership and control over the input dataset before releasing a privacy-preserving version of the input dataset. The proposed work seeks to identify the best version of the perturbed instance that can be released for analysis by considering a fine-tuned series of systematic steps, including: 1. Identify and categorize types of attributes based on privacy requirements. 2. Determine the level of privacy required based on the properties of the input dataset. 3. Generate multiple randomized versions of the input dataset 4. Identify the best perturbed version to publish based on utility, privacy, and attribution constraints.

[0038] Empirical results show that the disclosed method ensures that the final perturbed dataset provides sufficient utility and privacy and balances them appropriately by performing the above four steps.

[0039] Differential Privacy Differential privacy (DP) provides a mechanism to limit privacy leakage using two parameters: perturbation function ε (epsilon, also known as privacy budget) and δ (delta). The values ​​of these parameters determine the strength of privacy, i.e., the protection against linking of a dataset with additional datasets, enforced by a randomization (perturbation) algorithm (DP mechanism-M) for a particular dataset (D). ε provides insight into the degree of privacy loss that occurs during dataset disclosure. Therefore, ε should be kept at a lower level, for example, within the range 0 < ε ≦ 9 (below 10, two orders of magnitude). δ defines the probability of model failure. For example, if δ = 1 / 100 × n, the probability of failure is 1%. Therefore, δ should be kept at a very low level.

[0040] Definition of Differential Privacy We use dataset D and two adjacent datasets x and y (which differ by one record / person), where x and y are sets of records from population X,

number

[0041] Definition 1: Domain

number

number

[0042] Post-processing invariance of DP Post-processing invariance / robustness is the ability of a DP algorithm to remain robust against any additional computations on its output: additional computations / processing on the output do not weaken its original privacy guarantees, and therefore any result of post-processing on an ε-DP output remains ε-DP.

[0043] Fuzzy Inference System In some examples, the disclosed methods utilize fuzzy logic to derive a potential list of combinations of ε and δ for predefining privacy requirements according to an input dataset. That is, the methods calculate multiple values ​​of the parameters ε and δ of the perturbation function. Other ways besides fuzzy logic may be used to calculate the multiple values, such as decision trees, algebraic models, regression models, and others.

[0044] A fuzzy inference system (FIS) (fuzzy model) is derived through three sequential steps: (1) fuzzification, (2) rule evaluation, and (3) defuzzification. Fuzzification is the process of mapping distinct inputs to fuzzy values. For example, a specific input, such as temperature = 10°C, can be mapped to a fuzzy membership for low temperatures, generating membership values ​​ranging from 0 to 1. Next, the different levels of fuzzy membership values ​​generated by the inputs must be matched to a fuzzy output domain. This is done through rule evaluation in the FIS rule base. A fuzzy inference system consists of a list of linguistic rules (called the rule base) that allow for the evaluation of the different fuzzy membership levels generated during the fuzzification process. Defuzzification is the process of obtaining a quantifiable distinct output using rule evaluation and the aggregated membership degrees in the output parameter. The final distinct value is generated by applying a mechanism such as the centroid method (given in Equation 2) to the shape generated by the different membership levels of the output parameter.

number

[0045] Framework Figure 1 shows the primary modules of the disclosed framework (represented by squares), implemented as software modules, with arrows indicating the direction of data flow. In some examples, the method is controlled by a central party (e.g., government agency, hospital, bank) that has full ownership and control over the dataset. User role management over functional access may be used. However, in this example, the data curator has full access to the dataset at hand and the functionality of the algorithm that generates the privacy-preserving dataset.

[0046] Problem definition Let D be a dataset consisting of n tuples (rows) and m attributes (columns). Define S dataset as a vertical partition of D containing r∈m sensitive attributes. Let Dr be the S dataset and let D(mr) be the vertical partition of (mr). Using a differentially private algorithm (i.e., "perturbation function") M, perturb Dr to create a dataset with n tuples and r attributes.

number

number

number

number

[0047] This disclosure provides a unified framework-based approach that effectively answers all these questions.

[0048] Proposed Solution In one example, the dataset contains only non-categorical data. Let D be an input dataset with m×n attributes (m attributes and n tuples). The disclosed method automatically identifies a list of identifier attributes (ID) and quasi-attributes (Q). To protect against direct identification, the identifiers (ID attributes) are removed from the dataset. The dataset intended for publication after perturbation is formed by combining Q with the remaining vertical partition S, and is referred to as the QS dataset.

[0049] method FIG. 2a illustrates a computer-implemented method 200 for protecting an input dataset from linking with additional datasets. As noted above, this means, for example, protecting the dataset from linking individual rows with additional datasets that would allow for the identification of individuals in those rows. While some examples herein are provided with reference to the confidentiality of users and user data (such as patient data), the methods disclosed herein are equally applicable to other types of data. For example, it may be desirable to share operating machine parameters, such as turbine temperature, of an aircraft, but to protect this information from linking to additional turbine data that would allow for the identification of individual turbines. The method may be implemented as software and executed by a processor of a computer system, causing the processor to perform the steps of method 200.

[0050] In this sense, the processor calculates (201) multiple values ​​for one or more parameters (e.g., ε, δ) of a perturbation function (M). The perturbation function is configured to perturb the input dataset to protect the input dataset from linkage with additional datasets. The multiple values ​​of the perturbation function parameters indicate levels of protection against linkage with additional datasets. Note that there is no one-to-one relationship between the desired level of protection and ε, δ. In other words, there may be multiple pairs of ε, δ values ​​that provide the same, or substantially the same, level of protection against linkage. The problem then becomes how to select one of the pairs of values ​​from among seemingly equivalent candidates.

[0051] To address this issue, the processor generates multiple derived data sets from the input data set for different pairs of ε, δ values ​​202. This means that each of the multiple derived data sets is generated by applying a perturbation function to the input data set, and each of the multiple derived data sets is generated by using a different one of multiple values ​​of the ε, δ parameters of the perturbation function.

[0052] The processor calculates 203 a utility score for each of the plurality of derived data sets, as described below, that indicates the utility of the derived data set for the desired data analysis. Finally, the processor outputs 204 one of the plurality of derived data sets having the highest utility score.

[0053] FIG. 2b illustrates a computer system 250 for protecting an input dataset 251 from linking with an additional dataset 252. Note that in some examples, the input dataset 251 includes tabular data including rows and columns, such as data stored in a relational database, including SQL, Oracle, or others. The additional dataset 252 may be tabular data recorded in a relational database, but may also be stored in other forms. In particular, the additional dataset 252 may not be stored as rows and columns and may include only a small amount of information. For example, the additional dataset 252 may include only a single piece of information, such as a single record, that may be linked to one or more rows of the input dataset 251. This may allow for re-identification of rows from the input dataset if the input dataset is not sufficiently protected from linking.

[0054] Computer system 250 includes a custodian computer 253 having a processor 254, a program memory 255, and a communications port 256. Program memory 255 is a non-transitory computer-readable medium such as a hard drive, solid state disk, or CD-ROM. Software, i.e., an executable program, stored in program memory 255 causes processor 254 to perform the method of FIG. 2 a, i.e., to calculate multiple parameters of a perturbation function, generate multiple derived datasets from input dataset 251, and return the derived dataset with the highest utility score to requester computer 260.

[0055] The computer system 250 may be implemented in a cloud computing environment, such as a managed group of interconnected servers hosting a dynamic number of virtual machines, or through the use of general-purpose processors or application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). Parameters, values, variables, etc. are stored as digital data in program memory or separate volatile and non-volatile data memory.

[0056] 2a should be understood as a blueprint of a software program, which may be implemented step by step, such that each step in FIG. 2a is represented by a function in a programming language such as C++ or Java. The resulting source code is then compiled and stored as computer-executable instructions on program memory 255. In that sense, the program memory includes a data sharing module 257 that provides a derived dataset that protects input dataset 251 from association with further datasets 252.

[0057] The requester computer 260 sends a request to the custodian computer 253. To that end, the requester computer 260 also includes a processor 264 and a program memory 265. The processor 254 executes program code stored on the program memory 265 to request a data set from the custodian computer 253. In some embodiments, the requester computer 260 registers with the custodian computer 253 and authenticates itself. Through this authentication process, the custodian computer 253 can determine that the requester computer 260 meets a level of data protection. For example, the requester computer 260 has a demonstrated ability to keep any data sets confidential and prevent linking through access control, for example. In that case, the level of protection against linking implemented by the custodian computer 253 may be lower. In another example, the requester computer 260 is not authenticated, and therefore the custodian computer 253 assumes that the requester computer 260's purpose is to link the data sets with additional data and re-identify the records. In that case, there is a higher level of protection for the link. In that sense, there is a rule-based or hierarchical system that determines the level of protection based on the requester. Thus, the level of protection may be request-based, as it may include the identity and certification status of the requester computer 250.

[0058] Identifying ID and Q attributes An attribute is considered an identifier attribute (ID) if each field of that attribute is unique, leading to the unique identification of each record in the input dataset and enabling direct linking to sensitive information. If a particular attribute Puq (see Equation 3) is greater than a uniqueness threshold Tuq (e.g., 0.95), the processor considers it to be an ID attribute and removes it from the dataset. Thus, the lower the value of Tuq, the more stringent the selection of ID attributes.

number

[0059] Identify the initial tuple distribution of the dataset, allowing M to maintain the data distribution in QS The disclosed method applies Algorithm 1 below to generate status labels on tuples that classify them into specific clusters after performing mode imputation (mode imputation is used to accommodate both categorical and non-categorical data). This step allows the disclosed method to identify the tuple distribution of the original dataset and enables M to generate a perturbed version that resembles the data distribution of the original dataset. In some embodiments, the processor uses the k-means algorithm and silhouette analysis to identify optimal clustering dynamics for the input dataset. If the input dataset is a categorical dataset and each tuple has a class label, this step is not used because the class label represents the tuple distribution.

number

[0060] Identification Q attribute A set of attributes that in combination can uniquely identify a record is called a quasi-identifier (Q), which also leads to easy linking to auxiliary data and therefore carries the potential threat of leaking personal information.

[0061] Declare the Q attribute Selecting data-specific Q attributes is challenging because datasets from different domains may have different definitions of sensitive attributes. Thus, the sensitivity of a particular attribute depends on the context. If a human data curator has to select a Q attribute each time a method works on a particular dataset, they may unintentionally classify a sensitive attribute as one of the Q attributes. This can lead to accidental privacy leaks, as an attacker could potentially link a Q attribute to auxiliary knowledge that reveals the original value of the corresponding S attribute for a particular person.

[0062] This disclosure defines a global set of Q attributes (GQ attributes) that are common, most frequent, and common to a particular domain (e.g., commonly used by institutions using this disclosure). This approach allows for the selective selection of Q attributes common to different types of datasets and domains, making the selection of Q attributes simple, automated, and secure. At the same time, the selected Q attributes do not have an unacceptable level of indistinguishability within a given dataset. Therefore, the selected Q attributes are further refined through a process of extensively evaluating the sensitivity of the selected Q attributes in terms of the Privacy Indicator (PIF) metric, defined below. Again, note that the PIF indicates the degree of linkage of the input datasets.

[0063] Cell Surprise Factor (CSF) and Personal Information Factor (PIF) This disclosure defines a probabilistic measure called the Cell Surprise Factor (CSF), which is capped at 1 and provides a way to reason about how the indistinguishability of a record is affected by the participation of a particular attribute or set of attributes. The CSF of an attribute A (or set of attributes) is calculated according to Equation 4. The posterior distribution (DPo) is the conditional probability distribution (see Equation 5) of a record of A given a second attribute (B) record (or set of attributes). Thus, the CSF reflects changes or surprises in cell values ​​only, without interfering with other factors in the posterior. As a result, the CSF distribution provides a good representation of the indistinguishability of a particular attribute in D. If an attribute is indistinguishable, it also means that it is difficult to link this attribute to external data. On the other hand, if an attribute is distinguishable, it makes it easier to link the attribute to other data.

[0064] Here, the Personal Information Factor (PIF) is defined below to express the CSF distribution of an attribute through a single value bounded by [0,1].

number

number

[0065] definition, CSF=Abs(Prior(X)-Posterior(X)) (6)

[0066] Note that Posterior(X) is an upper bound because this method only looks at increasing indistinguishability. Thus, in most instances, Prior(X)≦Posterior(X).

[0067] Let xi be the csf value bin (bounded in [0,1]) of the attribute, and hi is the number of occurrences of each xi. Then,

number

[0068] Again, it is noted that the PIF is a scaled or weighted version of the CSF. In other words, the CSF represents the difference between the prior (unconditional) probability of an attribute relative to the posterior (conditional) probability of that attribute. The PIF represents a weighted combination of the CSFs using the number of occurrences. Thus, the PIF also indicates the degree of connectivity of the input dataset.

[0069] Applying perturbations to the QS dataset The perturbation process for the QS dataset is a four-step process: (1) further evaluating the Q attribute using the PIF; (2) refining (updating) the Q and S attributes based on the PIF analysis; (3) determining the privacy requirements (ε and δ) of the S dataset based on the PIF analysis; (4) performing perturbations on the S data and identifying locally optimal perturbed instances to be published;

[0070] In this sense, processor 254 partitions the input data set into Q and S partitions and applies the perturbation function only to the S partitions.

[0071] Further evaluation of Q attributes using PIF This step first generates PIF values ​​for all Q attributes in the Q dataset (QPIFi, where i represents the ith attribute). Next, PIF values ​​for all Q attributes in the QS dataset (QSPIFi) are calculated to determine the effect of S attributes on each Q attribute. The difference between QPIFi and QSPIFi for a particular Q provides evidence of how independent its data distribution is from S attributes. The inequality, ΔPIFi ≧ αQPIFi, can be used to determine that the change in PIF is α times QPIFi, where ΔPIFi = QSPIFi - QPIFi, α is the confidentiality coefficient. Therefore, maintaining α at 1 means that the PIF leakage from Qi in the QS dataset increases by exactly QPIFi. Since QSPIFi, QPIFi > 0, QSPIFi > QPIFi, and QSPIFi ≦ 1, we can obtain QSPIFi - QPIFi ≦ 1. Since 1 ≧ QSPIFi, QPIFi ≧ αQPIFi. Therefore, 1 ≧ αQPIFi. Therefore,

number

[0072] Furthermore, to satisfy this condition, QPIFi<5 must be satisfied. Therefore, Q attributes that satisfy the inequality ΔPIFi≧QPIFi are moved to the S dataset for perturbation. Once this step is complete, the method determines the privacy requirements of the S dataset by calculating the PIF (PIFThresh) of the QS dataset, as given in Equation 8. In the equation, QSMaxPIF is the maximum PIF value returned by the QS dataset. QMaxPIF is the maximum PIF of the refined Q dataset. As shown, PIFThresh takes into account the overall PIF leakage of the QS dataset as well as the additional PIF exposure caused by the Q dataset.

[0073] In another example, in the first step to determine the privacy requirements of the S dataset, the method calculates the PIF (PIFThresh) of the QS dataset using the following equation: In the equation, QSMaxPIF is the maximum PIF value returned by the QS dataset.

number

[0074] Establish a link between PIF and (ε, δ) With regard to enforcing differential privacy, the binding between PIF and (ε, δ) can be modeled as follows: The definition of (ε, δ)-differential privacy characterizes the probabilistic bounds of a randomized algorithm or statistical mechanism M. For each pair of adjacent data sets d and d' (which differ by a single individual's data), and for each possible subset of the output space S ⊆ Range(M), the model ensures that: P[M(d)∈S]≦eεP[M(d')∈S]+δ (8a) where P[M(d)∈S] denotes the probability that mechanism M produces an output in set S with input data set d.

[0075] Here, ε denotes a privacy parameter (privacy budget), and δ is a negligible quantity that represents the probability of a privacy mechanism potentially violating the ε-privacy condition. As ε approaches zero and δ is sufficiently small, a higher degree of privacy protection is granted. Therefore, we can define a privacy metric f(ε, δ) = (1 - exp(-ε)) + δ, which serves as a suitable gauge for quantifying the privacy level. Therefore, a decrease in the value of f(ε, δ) indicates enhanced privacy protection.

[0076] One property of differential privacy is postprocessing invariance, which implies that if a random mechanism M ensures (ε,δ)-differential privacy, then any postprocessing function g applied to the output of M also maintains (ε,δ)-differential privacy. Formally, if M ensures (ε,δ)-differential privacy, then a composition mechanism g≡M is also (ε,δ)-differentially private for all functions g.

[0077] In the non-interactive privacy-preserving data publishing paradigm, a data curator generates a differentially private version of a dataset D using a differentially private mechanism M. In this setting, f(ε,δ) acts as an upper bound on the privacy loss, ensuring that the privacy loss does not exceed (1-exp(-ε))+δ.

[0078] Looking at a particular attribute A∈D, a "Privacy Information Factor" (PIFA) can be defined, which quantifies the attribute-specific level of indistinguishability. For each attribute A, ΔA is defined as the increase in indistinguishability, which can be expressed as: ΔA=Posterior(A)-Prior(A) (9a)

[0079] The relationship between PIFA and ΔA is given as follows:

number

number

[0080] Using the PIFA of each attribute, the privacy measure fA is introduced as follows: fA(PIFA, δ)=PIFA+δ (11a)

[0081] As a result, the privacy measure for the entire dataset D can be derived using the maximum private information factor (PIFThresh) across all attributes of D. Thus, the privacy measure for a dataset can be defined as: fD(ε,δ)=PIFThresh+δ (12) fD(ε,δ) indicates an upper bound on the privacy loss when a dataset is published, providing a quantitative control mechanism to balance data utility and privacy protection.

number

[0082] Determining the Privacy Parameters (ε and δ) for S Dataset Perturbation In this disclosure, the values ​​of the parameters of the perturbation function are calculated based on the PIF. In particular, this disclosure uses a fuzzy inference system (FIS) to determine the boundaries of ε and δ for the S dataset based on the PIFThresh. The higher the value of the PIF (PIFThresh), the higher the indistinguishability of the QS dataset. Therefore, a high value of the PIF indicates that the S data requires high privacy and a high level of perturbation. This disclosure provides a fuzzy inference system between the PIF, ε, and delta to fit this relationship. All three fuzzy variables have three membership functions (low, medium, and high) that represent three levels of value range. All three membership functions have a Gaussian shape and range to accommodate a larger value range and a smooth transition from one membership level (function) to another (see FIG. 1). The mean (μ) and standard deviation (σ) for low, medium, and high are (μ=0, σ=1), (μ=0.5, σ=1), and (μ=1, σ=1), respectively.

[0083] Figure 3a shows the fuzzification of all three variables (ε, δ, and PIF). In this plot, the y-axis (degree of membership) quantifies the degree of membership of the corresponding input (ε, δ). Next, the method establishes a fuzzy rule base (a set of linguistic rules) that provides the basis for fuzzy inference. Equation 9 shows the proposed FIS rules. As shown in the equation, the rules are defined using IF-THEN transformations (e.g., IF(ε=MEDIUM AND δ=HIGH) THEN(PIF=MEDIUM)). The rule evaluation step of the FIS combines the fuzzy conclusions into a single conclusion by inferring the fuzzy rule base. In this step, a MAX-MIN (OR for MAX, AND for MIN) operation is applied to the rules. The minimum value between each membership level is considered for each rule, whereas the maximum fuzzy value of all rule outputs is used for the value conclusion. [Table 1]

[0084] Figure 3b illustrates the rule surface between three fuzzy variables. As shown on the rule surface, higher PIF values ​​correspond to lower values ​​for ε and δ. The final step in FIS is defuzzification based on the rule aggregation shape of the output function. The method uses a centroid-based technique to obtain the final defuzzified output value, where x = output and μx = degree of membership of x. As illustrated on the fuzzy rule surface (see Figure 3), a single PIF value corresponds to the set of (ε, δ) combinations.

[0085] Applying perturbations to the S data set In some embodiments, the disclosed method performs z-score normalization on the S dataset before perturbation to ensure that all S attributes are equally important and that the perturbation is normalized across the dataset. The method then generates a list of (ε and δ) combinations of corresponding PIFThresh for the input dataset. For a given (ε and δ) choice, the method performs the perturbation across the S dataset to generate a predetermined number of perturbed instances that resemble the data distribution provided herein. Each perturbed version is then min-max rescaled to return to the min-max values ​​of the original attributes and merged with the Q dataset to generate the perturbed QS dataset.

[0086] Usability analysis of perturbed instances The usefulness can be measured based on any measure, such as accuracy, precision, recall, and ROC area (KL divergence in a general scenario) normalized within [0,1]. Let KLx be the attribute of the perturbed instance DPi.

number

number

[0087] Definition 2 (Utility Loss-Ug)

number

[0088] In another example, the usefulness is measured based on any measure, such as accuracy, precision, recall, and ROC area (KL divergence in a general scenario), normalized within [0,1].

number

[0089] The effectiveness of the perturbation is evaluated by the normalized residual linking leakage, PN, and the ε-threshold, Tε, set by the OptimShare curators. If PN is too high, the dataset is not suitable for publication, and if Tε > ε,

number

[0090] The effectiveness loss (El) of a perturbed dataset is defined as a weighted measure of Ul and PN, calculated as El = CUL + (1-C)PN, where C determines whether to emphasize pegging protection (high C) or utility preservation (low C). The range of El depends on the PN and Ul values. For low PN and low Ul: El is [0,C]. For high PN, low Ul: El is [C,1]. For low PN, high Ul: El is [1-C,1]. For high PN, high Ul: El is [C,1]. In our study, we set C to 0.5 to treat residual pegging leakage and utility as equally important.

[0091] Privacy Analytics Once the perturbed instances of the input dataset are generated, the corresponding instances are checked for their vulnerability to data linking risk by evaluating the similarity between the tuples of the original and perturbed instances. This disclosure provides a threat model that addresses the worst-case scenario of the linking risk by assuming that an attacker has perfect knowledge of the Q attributes in the perturbed QS dataset.

[0092] Threat Model: The attacker has complete knowledge of the Q attributes (e.g., record order, attribute domain). This assumption leads to a worst-case linking risk by allowing the attacker to explore the linking degree of records through the Q attributes based on tuple similarity. The gained knowledge can then be used by the attacker to derive personal sensitive data.

[0093] This disclosure defines a similarity group SGk, which is a group of records in the QS dataset for which all Q records are the same. For each similarity group (SGk), the cosine similarity between the original S attributes and the perturbed S attributes of each record (ri) is

number

[0094] Definition 3 (Record Linkage Degree) Let R be the set of all rows in the perturbed (P) and original (D) datasets. Let (qα,sα)∈SG if qα=qβ for some α,β∈R and q∈Q. For each SGk∈SG,

number

number

number

number

number

[0095] For any α,β∈R such that qα=qβ for some q∈Q, the probability that (qα,sα) and (qβ,sβ) are in the same similarity group and that (qα,sα) are linkable is small.

[0096] Proof. Consider D as the original dataset with n tuples and m attributes. Define S and Q as the sets of sensitive and non-sensitive attributes in D, respectively. Assume that the attacker has perfect knowledge of Q in the perturbed dataset Dp.

[0097] The record linkage degree can be defined as follows: Consider R as the set of all records in D and Dp. If qα=qβ for some q∈Q and α,β∈R, then (qα,sα) and (qβ,sβ) are part of the same similarity group SG. The cosine similarity between the original and perturbed S attributes of each record ri in SGk

number

number

number

number

[0098] ε-differential privacy is satisfied if, for any data sets D1 and D2 that differ by at most one record, and any result o of a randomized algorithm M, the following inequality holds:

number

[0099] Take D1 as the original dataset and D2 as a dataset identical to D1 but with a modified confidentiality attribute in one record. Then, we show that ε-differential privacy can be applied and the attacker's probability of successful record matching is minimal.

[0100] Calculate the probabilities of the numerator and denominator of the inequality. The numerator probability is the chance that Dp contains the record (qα, sα) in the same SG as (qβ, sβ), and (qα, sα) are linkable. This is as follows:

number

[0101] For the denominator, the probability is the chance that Dp contains the record (qα,sβ) in the same SG as (qβ,sβ) and that (qα,sβ) is linkable.

number

[0102] Substituting into one of the equations above, we get:

number

[0103] This suggests that the attacker's probability of successful record matching is bounded, satisfying the ε-differential privacy requirement.

[0104] The disclosed method satisfies ε-differential privacy when the following inequality holds:

number

[0105] Proof. The above proof demonstrates that the numerator and denominator of the above equation are small, indicating that the probability that records within a similarity group are linkable is minimal. This requires verifying that perturbations to the confidentiality attribute of Dp are sufficient to thwart successful record linking by an attacker.

[0106] This can be achieved by ensuring that the cosine similarity between the original and perturbed confidentiality attributes of all Dp records is minimal. A lower cosine similarity makes record linking more difficult for an attacker, as it reduces the probability of record linking. Compliance with the privacy budget requires negligible changes in the probability of certain outcomes when records are added or removed, which can be achieved by applying DP noise to the confidentiality attributes during perturbation.

[0107] A sufficiently small cosine similarity between the original and perturbed attributes can be upper bounded using record linkage (Definition 3), which computes the cosine similarity for each dataset record. Complying with the privacy budget involves limiting the change in the probability of a particular outcome when adding or removing records.

[0108] Given two records (q1, s1) and (q2, s1') with the same quasi-identifier and confidentiality attributes s1 and s1' (where s1' is a perturbed version of s1 generated using an (ε,δ)-differential private generator), the cosine similarity between the original and perturbed confidentiality attributes can be calculated, indicating the small change in the probability of a particular outcome due to the addition or deletion of a record.

[0109] The cosine similarity between s1 and s1' is calculated as follows:

number

[0110] The Cauchy-Schwarz inequality can be used to show that: s1·s1'≦|s1||s1'|

[0111]

number

[0112]

number

[0113] Therefore, the cosine similarity between the original and the modified sensitive attributes is upper bounded by a value that adheres to the privacy budget ε, which ensures that the disclosed method satisfies ε-differential privacy.

[0114] Effectiveness analysis of perturbation and thresholding Let Tε be the threshold ε set by the curator. The normalized privacy leakage PN is defined according to Equation 11. For PN, the corresponding dataset is not considered for publication.

number

[0115] The effectiveness loss (E1) of a perturbed dataset is defined as a weighted metric of normalized privacy leakage and utility loss, as given in Equation 12. In one example, C is set to 0.5, treating both leakage (based on linkage) and utility equally. El = CUL + (1 - C)PN (12)

[0116] Proposed algorithm 4a and 4b illustrate an example algorithm as an algorithm flow of the steps for generating a privacy-preserving (perturbed) dataset, which shows how the disclosed method integrates the steps mentioned in the above sections in generating a privacy-preserving dataset.

[0117] result This section empirically demonstrates how the disclosed method derives optimally perturbed privacy-preserving datasets that are made public. We first show the dynamics of intermediate steps, followed by the dynamics of multiple perturbed instances of the input dataset. In this experimental evaluation, we used a MacBook Pro-2019 computer with an M1 Max and 32 GB of RAM for experiments on the dataset. For datasets with a larger number of tuples, we used one 112 Dual Xeon 14-core E5-2690 v4 Compute Node (with 256 GB of RAM) from the CSIRO Bracewell HPC cluster. [Table 2]

[0118] Experimental configuration During the experiments, the primary parameters of the algorithm were set with the following values: Tid = 0.95, Gq = ['ZIP Code', 'State', 'Country', 'BPQ020', 'RIAGENDR', 'ALQ120Q', 'LBXTC', 'Pregnancy', 'Age', 'Gender'], cnrange = [2, 3, 4, 5, 6, 7, 8], Uε = 8, Pl = 0.01%, TNε, δ = 12, TS = 4, t = 4, A = "Classification-GaussianNB", Ce = 0.5, ET = 0.8. These settings were kept constant throughout all experiments to maintain uniform experimental settings for unbiased results. DP-WGAN (Private Wasserstein GAN with Noisy Gradient Descent Moment Accountant) was used as the data perturbation technique for S data perturbation.

[0119] In other experiments, the primary parameters were set as follows: Tε = 8, Pl = 0.01% (δ = (1 / (100 × number of rows in D)) × Pl), Tε,δ = 12, t = 4, A = "Classification-GaussianNB", C = 0.5, ET = 0.5. The global Q attributes used for each dataset are provided in Figure 10. All settings remained constant across all experiments to ensure uniformity for unbiased results. For S data perturbations, DP-WGAN (focusing on non-categorical attributes) and PrivatePGM (focusing on categorical attributes) were used.

[0120] Dynamics of intermediate algorithmic steps This section evaluates the experimental dynamics of different thematic sections to understand the process underlying establishing a privacy-preserving dataset for public use. As discussed above, one of the components of the disclosed method is determining privacy requirements. This is done through PIF analysis, as explained above. As shown in Figure 5a, the input dataset exhibits extreme CSF values ​​(represented as dark) for certain attributes (e.g., BMXBMI, BMXHT), whereas certain other attributes, such as BPQ020, exhibit lower CSF values ​​(represented as light). This is due to the introduction of BMXBMI, which significantly reduces the overall indistinguishability of tuples in the dataset. However, BPQ020, among other attributes in the dataset, has a much smaller impact on reducing the indistinguishability of tuples. Therefore, a comparison between Figures 5a and 5b provides a clear indication of the intuition behind PIF value generation. As shown in Figure 5b, higher PIF values ​​indicate a higher level of indistinguishability (or PIF leakage) compared to other attributes.

[0121] As shown in Figures 5a and 5b, a separate analysis of the Q attribute provides a better understanding of the Q attribute that generates the PIF value compared to the Q attribute introduced into the S attribute, as represented by the red bar in Figure 5b. It is clear that when attributes LBXTC and ALQ120Q are introduced into the S attribute, their PIF values ​​increase significantly.

[0122] Figure 8 shows the CSF and PIF dynamics of the refined set of Q attributes. As illustrated by the plot, the disclosed method identified that LBXTC and ALQ120Q should be removed from the set of Q attributes because they leak too much information to be exposed without any perturbation. Therefore, LBXTC and ALQ120Q are automatically considered confidential attributes and moved to the set of S attributes. As shown in the plot (see Figures 6, 5a, and 5b), the refined Q attributes exhibit minimal data indistinguishability and produce greater uniformity among the tuples of the refined Q dataset. This result, in turn, supports the application of fewer perturbations to the S dataset compared to the previous unrefined Q attribute set.

[0123] Figure 7 shows the variation in the utility and effectiveness of the 12 datasets generated for the 12 combinations of ε,δ (TNε,δ=12). As shown in Figures 7a and 7b, the utility and effectiveness of the datasets are almost similar. This is because the corresponding datasets generate normalized privacy leakage (PN) that is much lower than the utility values. This also suggests that the disclosed method effectively refined the Q attribute so that the datasets can still maintain lower privacy leakage. To investigate the dynamics of utility and privacy in privacy-preserving datasets, we forced the disclosed method to stop refining the Q attribute.

[0124] Figure 8 shows that the dynamics of effectiveness are different from those of usefulness. This is because, as previously identified, due to high PIF leakage from all Q attributes, datasets tend to leak more information in certain scenarios. Here, PN has more influence on the effectiveness evaluation of the generated dataset, and therefore, the effectiveness plots show a relatively different pattern, as shown in Figure 5.

[0125] Figure 7 shows the utility variation of the intermediate datasets generated under four different rounds of data perturbation. From the bar graph, it is clear that the utility is not stable but changes under different rounds of perturbation. This demonstrates the importance of the systematic framework disclosed herein in determining the best version of a dataset to release by considering multiple factors, such as utility and privacy.

[0126] implementation We implemented two versions of the disclosed method (using Python 3.8): a server-based version for large-scale setups and a standalone version for single-computer use. Figure 9 shows an overview of the server-based system design with three user roles: a curator (data custodian), an operator (administrator), and a data user, each with distinct privileges. The curator owns and manages the original dataset, applies and audits data perturbations, and publishes the perturbed dataset for data users. As an administrator, the operator manages the algorithms while restricting access to the original dataset. Data users consume the perturbed dataset approved by the curator. The system ensures security and data privacy by allowing the dataset owner to exercise exclusive control and isolating the server from external access. This implementation uses Docker containers to store privacy-preserving algorithms for scalability and continuous integration and continuous deployment (CI / CD). The dataset manager then pushes the published dataset to the publishing system, and data users can only access the approved perturbed dataset.

[0127] It will be understood by those skilled in the art that numerous variations and / or modifications may be made to the above-described embodiments without departing from the broad general scope of the present disclosure, and the present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive.

Claims

1. 1. A computer-implemented method for protecting an input dataset from linking with further datasets, the method comprising: calculating a plurality of values ​​of one or more parameters of a perturbation function, the perturbation function configured to perturb the input dataset to protect the input dataset from linking with further datasets, each of the plurality of values ​​of the one or more parameters of the perturbation function indicating a level of protection against linking with further datasets; generating a plurality of derived data sets from the input data set, each of the plurality of derived data sets is generated by applying the perturbation function to the input data set; generating, wherein each of the plurality of derived data sets is generated by using a different one of the plurality of values ​​of the one or more parameters of the perturbation function; calculating a utility score for each of the plurality of derived data sets, the utility score indicating the utility of the derived data set for a desired data analysis; and outputting the one of the plurality of derived data sets having the highest usefulness score.

2. the method further comprising receiving a request for the data set from a requestor; The method of claim 1 , wherein the level of protection is based on one or more of the requestor or data in the request.

3. The method of claim 1 or 2, wherein calculating the plurality of values ​​of the one or more parameters of the perturbation function is based on a proximity index (PIF) of the input data set.

4. The method comprises: Calculating a plurality of cell surprise factors (CSFs), each CSF representing the indistinguishability of an attribute in the input data set; The method of claim 3 , further comprising: calculating the coefficient indicating the degree of connectivity of the input data set by combining the plurality of CSFs.

5. The method comprises: further comprising partitioning the input dataset into a first partition of quasi-identifiers and a second partition of sensitive data; The method of claim 4 , wherein the perturbation function is applied only to the second partition.

6. The method comprises: calculating the coefficient indicating the degree of association of the second partition with an attribute of the first partition; selectively adding the one attribute of the first partition to the second partition based on the calculated coefficient; The method of claim 5 , wherein the perturbation function is applied only to the second partition that includes selectively added attributes from the first partition.

7. 7. The method of claim 4, further comprising: performing a fuzzy interaction using the coefficients indicative of the degree of connectivity of the input data sets to determine the plurality of values ​​of the one or more parameters.

8. The method of claim 7 , wherein implementing the fuzzy intervention is based on a fuzzy membership function for the coefficient indicating the degree of linkage and each of the one or more parameters of the perturbation function.

9. 10. A method according to any one of the preceding claims, wherein the degree of linkage is measured in terms of differential ε, δ privacy, and the one or more parameters of the perturbation function are ε and δ.

10. 10. A method according to any one of the preceding claims, wherein the method further comprises removing identifier attributes from the input data set.

11. Calculating the utility score comprises: calculating a distribution difference between the input data set and the derived data set; and outputting the one of the derived data sets having the highest distribution difference.

12. Calculating the utility score comprises: calculating the accuracy of the desired data analysis for the derived data set; and outputting the one of the derived data sets having the highest accuracy.

13. The method comprises: applying a threat model to the derived dataset with the highest usefulness score to evaluate the similarity between tuples of the input dataset and the derived dataset; 10. The method of any one of the preceding claims, further comprising selectively blocking the output based on the assessment of the degree of similarity.

14. 10. The method of any one of the preceding claims, wherein calculating the utility score is based on utility loss and privacy leakage.

15. The method of claim 12 , wherein the utility score is a weighted sum of utility loss and privacy leakage.

16. 14. The method of claim 12 or 13, wherein the method selectively blocks output of the derived data set upon determining that the weighted sum of utility loss and privacy leakage is above a predetermined threshold.

17. Software which, when executed by a computer, causes the computer to carry out a method according to any one of the preceding claims.

18. A computer system comprising a processor programmed to carry out the method according to any one of claims 1 to 16.

Images