A system for designing and implementing privacy-preserving AI using privacy regulations within the intelligence pipeline

Abstract

Data including information corresponding to a set of users can be received. A privacy protection protocol to be applied to the data can be identified. A subset of the data can be identified as personally identifiable information (PII) data, the subset including a set of PII attributes. The PII attributes can be divided into categories based on a format of data fields in the PII attributes. The processed PII data can be combined with non-PII data to create processed client data. A determination can be made to add noise to a portion of the processed PII data. An amount of noise can be determined based on the privacy protection protocol. The amount of noise can be added to a portion of the processed PII data to generate protected data. A machine learning model can be trained using the protected data.

Description

[Technical Field]

[0001] Priority claims This application claims priority to U.S. Patent Application No. 17 / 932,938, entitled "SYSTEMS FOR DESIGN AND IMPLEMENTATION OF PRIVACY PRESERVING AI WITH PRIVACY REGULATIONS WITHIN INTELLIGENCE PIPELINES," filed September 16, 2022, which is incorporated herein by reference in its entirety.

[0002] Technical Field This disclosure relates generally to data privacy preserving protocols, and more particularly (though not necessarily exclusively) to implementing privacy-preserving artificial intelligence. [Background technology]

[0003] background The data platform for a client can provide tools for identifying users or user devices that can accept particular types of communications across multiple user experience streams. This identification of users or user devices can include processing historical and / or current data corresponding to multiple users or user devices, taking into account details corresponding to the communication type. As data from multiple systems is resolved in the data platform, privacy protection can become important (e.g., due to applicable privacy regulations and / or client-identified privacy goals).

[0004] For example, privacy regulations and / or goals may define what constitutes personally identifiable information (PII), which may be obscured before the data is stored, processed, used, and / or transmitted. PII data may include any information that can be used to distinguish users from one another and that can be used to de-anonymize previously anonymous data. Examples of PII may include social security numbers, passport numbers, telephone numbers, credit card numbers, addresses, patient identification numbers, email addresses, etc. However, what constitutes PII may be context-specific. For example, assigning a given user to an income bracket may be insufficient to identify the user in most zip codes, but may be useful for identifying a particular person in one or more particular zip codes. Furthermore, the regulations and / or priorities of various entities regarding data privacy may change over time and vary from region to region (e.g., because privacy regulations may change over time and from region to region). Summary of the Invention [Problem to be solved by the invention]

[0005] When an entity collects PII, it may be important to implement protocols to ensure that the storage and transmission of PII is controlled to prevent unauthorized access. However, it may also be important to control the use of PII. For example, in some cases, when a dataset having PII is used to train a machine learning (ML) model, it may be possible for the model to be interrogated in a way that allows an attacker to predict PII (e.g., by examining the outputs generated in response to particular inputs). [Means for solving the problem]

[0006] overview Techniques (e.g., methods, systems, non-transitory computer-readable media storing code or instructions executable by one or more processors) are provided for the design and implementation of privacy-preserving AI with privacy regulations within an intelligence pipeline. Various embodiments are described herein, including methods, systems, programs executable by one or more processors, non-transitory computer-readable storage media storing code or instructions, etc.

[0007] One embodiment relates to a computer-implemented method including: receiving client data from a client system, the client data including information corresponding to a set of users and / or a set of user devices; identifying a privacy-preserving protocol to be applied to the client data; identifying a particular subset of the client data as personally identifiable information (PII) data, the particular subset including a set of PII attributes; dividing the PII attributes into a plurality of predetermined categories based on a format of a data field in the PII attributes; for each predetermined category of the plurality of predetermined categories, selecting a processing technique from a plurality of processing techniques based on the predetermined category and the privacy-preserving protocol; processing the PII data corresponding to the predetermined category using the selected processing technique; combining the processed PII data corresponding to the plurality of predetermined categories with non-PII data to create processed client data; determining, based on previous input from the client system and / or based on the processed PII data, to add noise to at least a portion of the processed PII data; determining an amount of noise based on the privacy-preserving protocol;

[0008] In some embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable storage medium that includes instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0009] In some embodiments, a computer program product is provided that is tangibly embodied in a non-transitory machine-readable storage medium and includes instructions configured to cause one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0010] The present disclosure is described in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0011] [Figure 1] 1 shows a flow diagram illustrating an exemplary process for performing privacy-preserving AI, according to some embodiments of the present invention. [Figure 2] 1 illustrates a database environment for running privacy-preserving AI, in accordance with at least some embodiments of the present invention. [Figure 3] FIG. 1 illustrates the flow of client data through a privacy-preserving protocol, according to at least some embodiments. [Figure 4] FIG. 2 is a flow diagram illustrating an exemplary process for protecting the privacy of client data by enforcing privacy rules, according to some embodiments of the present invention. [Figure 5] FIG. 10 is a block diagram illustrating separation of PII and non-PII data by a PII identifier module, according to an embodiment. [Figure 6] 1 is a graph illustrating the dependency of the root mean square error of a recommender algorithm when noise is added to the ML model training data, according to an embodiment. [Figure 7]1 is a graph showing the relationship between the mean absolute error of a recommender algorithm and the value of epsilon when applying a privacy budget to training data for an ML model. [Figure 8] 8 is a graph illustrating accuracy rates of ML models as a function of epsilon 802 using an algorithmic noise addition approach in an ML model privacy protocol, according to an embodiment. [Figure 9] 9 shows a simplified diagram of a distributed system 900 for implementing an embodiment. [Figure 10] FIG. 1 is a schematic block diagram of a cloud-based system environment in which the functionality described herein can be provided as a cloud service, according to an embodiment. [Figure 11] 11 illustrates an exemplary computer system 1100 that can be used to implement certain embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0012] In the accompanying figures, similar components or features may have the same reference label. Furthermore, various components of the same type may be distinguished by following the reference label with a dash and a second label that distinguishes between the similar components. When only a first reference label is used in the specification, the description is applicable to any of the similar components having the same first reference label, regardless of the second reference label.

[0013] Detailed Description In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of certain embodiments. It will be apparent, however, that various embodiments may be practiced without these specific details. The figures and descriptions are not intended to be limiting. The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0014] Some embodiments of the present invention relate to systems and techniques for generating and executing privacy-preserving protocols through privacy-preserving artificial intelligence (AI). In particular, incoming data can be parsed to distinguish data corresponding to multiple predetermined categories (e.g., categorical text data, categorical numeric data, high-cardinality text, high-cardinality numeric data, continuous numeric data, and text). In some examples, a different type of privacy-preserving protocol is executed for each category of data. For each type of data, a technique can be selected for transforming PII within the category (e.g., if corresponding PII transformation criteria are met).

[0015] 1 shows a flow diagram illustrating an example process 100 for performing privacy-preserving AI, according to some embodiments of the present invention. Block 102 includes receiving client data from a client system via a data platform. The client system may include at least one of a mobile device, a cloud storage platform, a personal computer, a laptop, a computer cluster, a server, a server cluster, etc. The client data may be received via an interface generated by the data platform. In some examples, the interface may identify regional and client privacy regulations for the client data.

[0016] In block 104, a privacy protection protocol is defined for PII in the client data. In some examples, the client may define the privacy protection protocol using a configuration screen. Block 104 may include enabling a toggle switch to allow the client to mask, obfuscate, or hide PII data using a variety of privacy options. The client may choose to implement different privacy protection protocols for different categories of PII data. In some examples, the privacy protection protocol may be determined by a server operator associated with the data platform, such as when the client selects a default setting for the privacy protection protocol or provides limited input to a configuration screen. The privacy protection protocol may be based on regional or client privacy regulations detected by the interface when the interface receives the client data. For example, the server operator may verify that the privacy protection protocol selected by the client complies with regional privacy regulations.

[0017] Because ML model training data may include PII data, a client may choose to include privacy settings for data associated with the ML model as part of a privacy protection protocol. In some examples, preprocessing of the ML model training data may be configured based on an ML privacy budget, a default value, etc., which may be set by the client. By adjusting the ML privacy budget, a client may define the degree to which PII data is obscured before entering the ML model. For example, noise may be added to the PII data. A low amount of noise may correspond to high accuracy in the ML model but may not provide adequate privacy protection for the PII data input to the ML model. If the amount of noise increases, privacy protection may be enhanced, but the model's accuracy rate may be reduced. In some examples, the interface may notify the client of the predicted accuracy level in the ML model based on the selected ML privacy budget. In some examples, noise may be added to the algorithm or output of the ML model according to the client's selection of the privacy budget. The client may adjust the ML privacy budget to tune the model accuracy rate and / or privacy. In some examples, the output of the ML model may include PII data, and the client may include privacy settings for the output in the privacy protection protocol.

[0018] At block 106, the process includes identifying PII data in the client data. In some examples, the client data is forwarded to a PII identifier module. The PII identifier module can identify which attributes contain PII and which are non-PII attributes. For example, this module can recognize that a set of nine digits indicates a Social Security number, or that a set of seven digits or a set of ten digits indicates a U.S. telephone number including an area code. The PII identifier module can identify PII attributes using a PII identifier program and an imported PII infotype recognition library. The data type divider module can divide the PII attributes into categories. The categories include categorical text, categorical numeric, high-cardinality text, high-cardinality numeric, continuous value, text, etc.

[0019] At block 108, the process includes processing the PII data using the selected processing technique. In some examples, the data type divider module can process PII attributes based on categories and based on privacy rules and heuristics associated with a privacy protection protocol supplied by the client. For example, there may be several processing techniques for processing PII attributes, including adding noise to the PII data in the ML model training data. In some examples, block 110 can include integrated masking, hashing, pseudonymization, and anonymization of the identified PII data based on the PII infotype.

[0020] At block 110, the process includes identifying an amount of noise based on a privacy protection protocol and an ML privacy budget. Adding noise to the ML training data can provide some protection for the ML model, but it can also reduce the accuracy rate of the ML model. Therefore, the amount of noise can be determined based on an ML privacy budget specified by the client, allowing the client to control this balance.

[0021] At block 112, the process includes adding an amount of noise to a subset of the combined PII and non-PII client data to generate protected training data. If data privacy is selected for a category of PII data by the client, random, Gaussian, or Laplace noise may be added to the category of data with PII attributes. Noise may not be added to non-PII client data included in the protected training data.

[0022] Block 114 of the process involves inputting the protected training data into an ML model. Once trained, the ML model can generate an output. In some examples, the output is returned to the client device.

[0023] FIG. 2 illustrates a database environment 200 for implementing privacy-preserving AI, in accordance with at least some embodiments of the present invention. The database environment 200 includes a client data platform 202 and multiple client systems 208. While three client systems are shown in FIG. 2, more or fewer client systems may be present in the database environment 200. Each client system of the multiple client systems 208 can receive client data from a user device 216. For example, client system A can receive data from user device A 216-a, user device B 216-b, user device C 216-c, and user device D 216-d. Client system B can receive data from user device E 216-e, user device F 216-f, and user device G 216-g. Client system C can receive data from user device H 216-a, user device I 216-i, user device J 216-j, user device K 216-k, and user device L 216-l. 2 shows twelve user devices 216, more or fewer user devices 216 may be present in database environment 200. Client data received from user devices 216 may include PII data.

[0024] Each client system of the plurality of client systems 208 can store client data in a user data store 212. Each client system of the plurality of client systems 208 can communicate with the client data platform 202. Each client system can send client-specific privacy rules 210 to the client data platform 202. In some examples, for each client system, the client data platform 202 can determine the client system's applicable regional privacy rules 204 by detecting the location of a user associated with the client. Detecting the user's location can involve extracting a field of information from data involved in communication with the user device 216. In some examples, the user's location can be extracted from home address data included in the client data. In some examples, the client data platform can receive information regarding the regional privacy rules 204 from the client.

[0025] The client data platform 202 can communicate with each client system in the plurality of client systems 208 to define a privacy protection protocol for PII data in each client system. In some examples, the client can define the privacy protection protocol using a configuration screen. The client can choose to implement different privacy protection protocols for different categories of PII data. In some examples, the client can define a privacy budget to provide privacy settings for client data associated with the client's ML model.

[0026] In some examples, the client data platform 202 can receive user data from a client, and a client-specific privacy protection program can be executed by the client data platform 202. The client data platform 202 can modify the privacy protection program 206 to create a client-specific privacy protection program that includes a privacy protection protocol and is based on client-specific privacy rules, applicable regional privacy rules, and a privacy budget. In some examples, a client system can receive the privacy protection program 206 from the client data platform 202, and the privacy protection program can be executed by the client system. Each client system can execute the privacy protection protocol using the client-specific protection program to maintain data security measures that comply with regional and client-specific privacy rules.

[0027] FIG. 3 illustrates the flow of client data through a privacy protocol, according to at least some embodiments. Client data includes data collected from users associated with clients. Data collected from users may include PII data. Users associated with clients may reside in various regions. Regions may include collections of various countries, individual countries, states, provinces, etc. Privacy protection policies for users' PII data may vary by region and industry. For example, a country may require clients to implement and maintain specific data security measures to protect the financial information of users who are citizens of that country. Client data may originate from user devices, which may include mobile devices, personal computers, laptops, smart watches, etc. Client data may be aggregated in a data repository associated with the client system, called a data warehouse 302. Client data may include both PII and non-PII attributes.

[0028] In some examples, client data from the data warehouse 302 can be exported to the client data platform 202. In the example shown in FIG. 3, the client data platform 202 is shown to include cloud storage 304 or an object storage (OS) or simple storage service (S3) budget. A PII identifier module 306 associated with the client data platform 202 can analyze the client data in the cloud storage 304. The PII identifier module 306 can identify which attributes include PII attributes and which include non-PII attributes using techniques such as a PII identifier program, a PII infotype recognition library, or a PII definition identified by the client. Identification of PII attributes can vary by organization, domain, country, and region, and such specificity may be reflected (for example) by program functions, libraries, or client-specified definitions. The PII identifier module 306 can be configured to incorporate various public or private PII infotype libraries. The PII infotype library can be stored within the client data platform 202, or the PII infotype library can be received from and associated with the client. Examples of PII infotype libraries include Presidio (Microsoft SDK), AWS Detect PII, and open source libraries.

[0029] The data type splitter module can split PII attributes into categories. Categories include categorical text, categorical numeric, high-cardinality text, high-cardinality numeric, continuous value, text, etc. For example, the data type splitter module can separate PII data with numeric attributes (numeric data) from PII data with categorical attributes (categorical data). Examples of numeric data can include zip code, salary, age, Social Security number, date of birth, etc. Examples of categorical data can include name, gender, city of residence, race, place of birth, etc. Categorical data can be identified using rule-based algorithms, such as entity recognition algorithms, such as named entity recognition (NER) algorithms. Pattern recognition techniques can be used to identify certain types of numeric values ​​(e.g., bank account data) or certain types of categorical data (e.g., national identification information).

[0030] A privacy protection protocol 308 can be performed on attributes in the client data. Because privacy protection policies may vary by region, the privacy protection protocol 308 can depend on the citizenship of a user associated with the client data. In some examples, identifying the privacy protection protocol 308 can include determining one or more geographic regions corresponding to the client system and retrieving one or more data protection constraints corresponding to the one or more geographic regions. The privacy protection protocol 308 can also depend on a client-specific privacy protection policy. The decision to perform the privacy protection protocol 308 can depend on whether the attribute of the client data is a PII attribute or a non-PII attribute. The privacy protection protocol 308 performed on the PII attribute can depend on the category of the PII attribute. For example, when numerical data is subject to a regional privacy protection policy, noise (e.g., Gaussian noise) can be added to the numerical data as data privacy (DP) preprocessing 316. In another example, the numeric data is not subject to a privacy protection policy, and a privacy protection protocol is not implemented for the notification data. In other words, when the numeric data is not subject to a privacy protection policy, noise is not added to the numeric data. If the privacy protection protocol 308 is not implemented for the first dataset (e.g., the first dataset includes only non-PII attributes), the first dataset can be used to train an ML model without undergoing (DP) preprocessing 316.

[0031] When a dataset having PII is used to train a machine learning (ML) model, it may be possible for the model to be interrogated (by examining the outputs generated in response to specific inputs) in a manner that allows an attacker to predict PII data. Preprocessing of ML model training data can be configured based on an ML privacy budget, a default value, etc., which can be set by the client. By adjusting the ML privacy budget, the client can define the degree to which PII data (or non-PII data) is obscured before input to the ML model. For example, noise can be added to the PII data. A low amount of noise may correspond to high accuracy in the ML model but may not provide adequate privacy protection for the PII data input to the ML model. If the amount of noise is increased, privacy protection can be strengthened, but the accuracy rate of the model may be reduced.

[0032] The ML privacy protocol 310 can be executed to process the first dataset even if the privacy-preserving protocol 308 was not executed for the first dataset. If the ML privacy protocol 310 is not selected for the first dataset, training data for an ML model using a standard ML algorithm can include the first dataset at block 312. Once trained, ML scoring can be performed, and the results of the standard ML model can be returned to the client via the data warehouse 302. If the ML privacy protocol 310 is selected for the first dataset, training data for an ML model using a privacy-preserving ML algorithm can include the first dataset at block 314. Once trained, ML scoring can be performed, and the results of the privacy-preserving ML model can be returned to the client via the data warehouse 302.

[0033] If the privacy protection protocol 308 is performed on the second data set (e.g., the second data set includes at least one category of PII attributes that are subject to the privacy protection policy), the second data set can undergo DP preprocessing 316. DP preprocessing 316 can include integrated masking, hashing, pseudonymization, or anonymization of the second data set based on the PII infotype. The type of preprocessing can depend on the category of PII attributes included in the second data set. For example, random, Gaussian, or Laplace noise can be added to numeric attributes, and categorical attributes can be encrypted.

[0034] An ML privacy protocol 318 can be executed on the second data set. If the ML privacy protocol 318 is not selected for the second data set, the training data for the ML model using a standard ML algorithm can include the second data set in block 320. Once trained, ML scoring can be performed, and the results of the standard ML model can be returned to the client via the data warehouse 302. If the ML privacy protocol 318 is selected for the second data set, the training data for the ML model using a privacy-preserving ML algorithm can include the first data set in block 322. Once trained, ML scoring can be performed, and the results of the privacy-preserving ML model can be returned to the client via the data warehouse 302.

[0035] 4 is a flow diagram illustrating an example process 400 for privacy protection of client data by enforcing privacy rules, according to some embodiments of the present invention. Block 402 includes data cleaning of the client data. Data cleaning may involve correcting incorrect, duplicate, or otherwise erroneous data from a data set in the client data. For example, data cleaning may involve syntax errors or formatting errors. As an example, a recorded date of birth of 13.03.92 may be changed to 03 / 13 / 92 to align or match other data in the data set.

[0036] Block 404 includes forwarding the client data to the PII identifier module 306 to identify PII attribute data and non-PII attribute data in the client data. The PII identifier module 306 can identify PII attributes in the client data using a PII identifier program and an imported PII infotype recognition library. The infotype recognition library can identify patterns in a data set that can be used to identify PII attributes. For example, in the United States, a five-digit data set may refer to a zip code, or data in the format (XXX)-XXX-XXXX (where X is any digit) may be recognized as a phone number. The PII identifier module 306 can be configured to support various public or private PII infotype libraries. The PII infotype libraries supported by the PII identifier module 306 can be periodically updated.

[0037] Block 406 involves forwarding the PII attribute data to a data type divider module. The data type divider module can divide the PII attributes into categories. Categories can include categorical text, categorical numeric, high-cardinality text, high-cardinality numeric, continuous value, text, etc. For example, the data type divider module can separate PII data with numeric attributes (numeric data) from PII data with categorical attributes (categorical data). Examples of numeric data include zip code, salary, age, Social Security number, date of birth, etc. In some examples, numeric data in a region can be identified by comparing the population distribution of a region. For example, if a set of PII data exhibits significant overlap with the population distribution (e.g., greater than 50% overlap), the PII data can be identified as numeric data. Categorical data can include name, gender, city of residence, race, place of birth, etc. Categorical data can be identified using entity recognition algorithms such as NER algorithms. Pattern recognition techniques can be used to identify certain types of numeric data (e.g., bank account data) or certain types of categorical data (e.g., national identification information).

[0038] Block 408 involves applying privacy rules to the numeric data and categorical data. Because privacy regulations may vary by country and region, the source of the client data may affect the privacy rules. In some examples, applying privacy rules to the numeric data involves converting the numeric data into noisy data by adding noise to the numeric data. The noise may include random noise, Gaussian noise, or Laplace noise. The privacy rules applied to the PII data may depend on the type. For example, categorical numeric data containing a limited number of values ​​to represent gender may not require as strict privacy steps as privacy protocol requirements for PII data containing a user's bank account number or social security number. In some examples, applying privacy rules to the categorical data involves converting the categorical data into encrypted data by adding encryption to the categorical data. The noisy data and encrypted data may be combined to form a privacy data set.

[0039] Block 410 involves training a machine model using the protected training data. For example, noisy data, encrypted data, and non-PII attribute data may be combined to form the protected training data. Once trained, the ML model may generate an output. In some examples, the output may correspond to a client-defined prediction, and the determined prediction may be returned to the client device. If PII attribute data is present in the output, privacy rules may be applied to the PII attribute output data.

[0040] 5 is a block diagram illustrating separation of PII data and non-PII data by the PII identifier module 306, according to an embodiment. The preprocessing module of the PII identifier module 306 can separate client data into multiple categories based on attribute type. The attribute types can be PII attributes and non-PII attributes. The categories based on attribute type can include text 502, categorical text 504, high cardinality text 506, continuous text 508, high cardinality numeric 510, and categorical numeric 512.

[0041] Categorical text 504 may include text associated with a fixed or limited number of possible values. An example of categorical text 504 may include the make of the vehicle a user drives. Another example of categorical text is the user's state of residence. High cardinality data may include highly unique or uncommon data that can be used to identify a user. Examples of high cardinality text 506 include email addresses, usernames, etc. Examples of high cardinality numeric values ​​510 include bank account numbers, social security numbers, telephone numbers, etc. Categorical numeric values ​​512 may include numeric values ​​assigned to a limited number of possible values. An example of categorical numeric values ​​may be limited numeric values ​​assigned to each possible gender or each possible blood type. Continuous text 508 data may include text assembled into sentences, which are organized into paragraphs, which are organized into sections, chapters, etc.

[0042] Various techniques can be used to identify PII data within a category based on attribute type. PII data can be identified within the text 502 category using ML modeling techniques 514. For example, ML modeling techniques 514 can be applied to a text column to search for PII attributes such as a person's name, organization name, and location. If the ML modeling techniques find a PII attribute in at least a threshold percentage (e.g., at least 80%) of the rows in the column, the column is identified as PII data. Additionally, privacy rule-based techniques 516 can be applied to identify PII data. For example, high-cardinality numeric 520 categories can be compared to distributions of PII attributes such as age, area population, or salary. If a distribution set of high-cardinality numeric 520 data exhibits statistically significant overlap with those distributions, the set can be identified as PII data. Pattern recognition 518 is another technique for identifying PII data such as passport numbers, tax identification numbers, national identification numbers, bank account numbers, etc.

[0043] FIG. 6 is a graph showing the dependency of the root mean square error of a recommender algorithm when noise is added to ML model training data, according to an embodiment. To protect the PII data used to train the ML model, strict privacy techniques for the ML model algorithm may be required. A list of potential threats to ML systems may include de-anonymization (re-identification) attacks, reconstruction attacks, parameter interference attacks, model inversion attacks, membership inference attacks, etc. Even when a dataset is anonymized, an attacker may use background knowledge to infer data through a de-anonymization or re-identification attack. An attacker can obtain external knowledge of the feature vectors of the ML model training data in a reconstruction attack. In a model inversion attack, an attacker may discover a user's submission of a new test sample to the ML model and the response generated by the ML model for the new test sample. In a membership inference attack, an attacker may gain access to the ML model output and, based on the identified output, identify whether a particular set of PII data was used to train the ML model.

[0044] Table 1 includes various methods by which PII data involved in ML models can be protected. Protection of data incorporated into ML models can be configured based on an ML privacy budget, default values, etc., which can be set by the client. By adjusting the ML privacy budget, the client can define the degree to which PII data is involved in the ML model. Noise can be added at various stages associated with the ML model. To protect PII data used to train ML models, in some examples, differential privacy techniques can be employed. Differential privacy refers to techniques implemented to ensure limits on how much information can be revealed due to participation in a database. Essentially, differential privacy places a limit on the amount of information an attacker can access.

[0045] [Table 1]

[0046] The constraints provided by differential privacy can be determined by a variable represented by the Greek letter epsilon (ε). The value of epsilon is roughly the number of information bits an analyst can determine about an individual from a database. Epsilon is a parameter that can be linked to the ML privacy budget. The value of epsilon can be selected based on the ML privacy budget. Smaller values ​​of epsilon result in a larger amount of added noise (i.e., a larger privacy budget). Therefore, smaller values ​​of epsilon are associated with stronger privacy protection, but with a reduced accuracy rate.

[0047] The graph in Figure 6 shows the relationship between the RMSE of a recommender algorithm and the value of epsilon when applying a privacy budget to the training data of an ML model. For comparison, the non-privacy value 604 of the RMSE when no noise is added is also shown. In this example, the non-privacy value 604 is 0.6854. As epsilon increases, the difference between the RMSE of the privacy data 602 and the non-privacy value 604 decreases. As the value of epsilon decreases, the difference between the privacy data 602 and the non-privacy value 604 increases.

[0048] 7 is a graph showing the relationship between the mean absolute error of a recommender algorithm and the value of epsilon when a privacy budget is applied to training data for an ML model. For comparison, the non-privacy value of mean absolute error 704 when no noise is added is also shown. In this example, the non-privacy value of mean absolute error 704 is 0.398. As epsilon increases, the difference between the mean absolute error of the privacy data 702 and the non-privacy value of mean absolute error 704 decreases. As epsilon decreases, the difference between the mean absolute error of the privacy data 702 and the non-privacy data 704 increases.

[0049] FIG. 8 is a graph illustrating the accuracy rate of an ML model as a function of epsilon 802 using an algorithmic noise addition approach in an ML model privacy protocol, according to an embodiment. In the algorithmic noise addition approach, the ML model may undergo several iterations to complete an ML model training phase. During the ML model training phase, model parameters may be optimized through attempts to reduce noise at each iteration. Various ML algorithms used may include logistic regression, naive Bayes, stochastic gradient descent, or deep neural networks. The graph in FIG. 8 shows the accuracy rate of the naive Bayes algorithm as epsilon is varied. As discussed above, the selected value of epsilon can affect the amount of noise added to the naive Bayes algorithm. For comparison, accuracy rate values ​​804 when no noise is added to the algorithm are also shown. The graph indicates that lower epsilon values ​​provide greater privacy protection, but result in a reduced accuracy rate.

[0050] 9 shows a simplified diagram of a distributed system 900 for implementing an embodiment. In the illustrated embodiment, the distributed system 900 includes one or more client computing devices 902, 904, 906, and 908 coupled to a server 912 via one or more communication networks 910. The client computing devices 902, 904, 906, and 908 can be configured to run one or more applications.

[0051] In various embodiments, server 912 may be adapted to run one or more services or software applications that enable the processing described in this disclosure.

[0052] In certain embodiments, server 912 may also provide other services or software applications, which may include non-virtualized and virtualized environments. In some embodiments, these services may be provided to users of client computing devices 902, 904, 906, and / or 908 as web-based or cloud services, such as under a software-as-a-service (SaaS) model. Users operating client computing devices 902, 904, 906, and / or 908 may in turn utilize one or more client applications to interact with server 912 and utilize the services provided by these components.

[0053] 9, server 912 may include one or more components 918, 920, and 922 that implement the functions performed by server 912. These components may include software components that may be executed by one or more processors, hardware components, or a combination thereof. It should be understood that a variety of different system configurations are possible that may differ from distributed system 900. Thus, the embodiment shown in FIG. 9 is an example of a distributed system for implementing the system of the embodiments and is not intended to be limiting.

[0054] According to the teachings of the present disclosure, a user can interact with server 912 using client computing devices 902, 904, 906, and / or 908. The client devices can provide an interface that allows a user of the client device to interact with the client device. The client devices can also output information to the user via the interface. Although FIG. 9 shows only four client computing devices, any number of client computing devices can be supported.

[0055] Computing devices can include various types of computing systems, such as portable handheld devices, general-purpose computers such as personal computers and laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computing devices can run various types and versions of software applications and operating systems (e.g., Microsoft Windows®, Apple Macintosh®, UNIX® or UNIX-like operating systems, Linux® or Linux-like operating systems such as Google® Chrome™ OS), including various mobile operating systems (e.g., Microsoft Windows Mobile®, iOS®, Windows Phone®, Android™, BlackBerry®, Palm OS®). Portable handheld devices can include mobile phones, smartphones (e.g., iPhone®), tablets (e.g., iPad®), personal digital assistants (PDAs), etc. Wearable devices can include Google Glass® head-mounted displays and other devices. The gaming systems may include various handheld gaming devices, Internet-enabled gaming devices (e.g., Microsoft Xbox® game consoles with or without Kinect® gesture input devices, Sony Playstation® systems, various gaming systems offered by Nintendo®, etc.), etc. The client devices may run a variety of different applications, such as various Internet-related apps, communication applications (e.g., email applications, short message service (SMS) applications), etc., and may use a variety of communication protocols.

[0056] Network 910 can be any type of network familiar to those skilled in the art and can support data communications using any of a wide variety of available protocols, including, but not limited to, TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (Systems Network Architecture), IPX (Internet Packet Exchange), AppleTalk®, etc. By way of example only, network 910 can be a local area network (LAN), Ethernet®, token ring, wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., a network operating under any of the Institute of Electrical and Electronics Engineers (IEEE) 1002.11 protocol suite, Bluetooth®, and / or other wireless protocols), and / or a network based on any combination of these and / or other networks.

[0057] Servers 912 may be comprised of one or more general-purpose computers, dedicated server computers (including, by way of example, PC (personal computer) servers, UNIX servers, midrange servers, mainframe computers, rack-mounted servers, etc.), server farms, server clusters, or any other suitable arrangement and / or combination. Servers 912 may include one or more virtual machines running virtual operating systems or other computing architectures involving virtualization, such as one or more flexible pools of logical storage devices that may be virtualized to maintain virtual storage devices for the servers. In various examples, servers 912 may be adapted to run one or more services or software applications that provide the functionality described in the foregoing disclosure.

[0058] The computing system of server 912 may run one or more operating systems, including any of those mentioned above, as well as any commercially available server operating system. Server 912 may also run any of a wide variety of additional server and / or middle-tier applications, including HTTP (Hypertext Transport Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, JAVA servers, database servers, etc. Exemplary database servers include, but are not limited to, those commercially available from Oracle®, Microsoft®, Sybase®, IBM® (International Business Machines), etc.

[0059] In some implementations, server 912 may include one or more applications for analyzing and aggregating data feeds and / or event updates received from users of client computing devices 902, 904, 906, and 908. By way of example, the data feeds and / or event updates may include, but are not limited to, Twitter® feeds, Facebook® updates, or real-time updates received from one or more third-party information sources and continuous data streams, which may include real-time events related to sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc. Server 912 also may include one or more applications for displaying the data feeds and / or real-time events via one or more display devices of client computing devices 902, 904, 906, and 908.

[0060] The distributed system 900 may also include one or more data repositories 914, 916. These data repositories, in some embodiments, may be used to store data and other information. For example, one or more of the data repositories 914, 916 may be used to store data or information generated by and / or used for the processes described herein. The data repositories 914, 916 may reside in a variety of locations. For example, a data repository used by the server 912 may be local to the server 912 or may be remote from the server 912 and communicate with the server 912 via a network-based or dedicated connection. The data repositories 914, 916 may be of different types. In some embodiments, the data repository used by the server 912 may be a database, for example, a relational database such as those provided by Oracle Corporation® and other vendors. One or more of these databases may be adapted to allow data to be stored, updated, and retrieved from the database in response to SQL-formatted commands.

[0061] In some embodiments, one or more of the data repositories 914, 916 may also be used by an application to store application data. The data repositories used by an application may be of different types.

[0062] In some embodiments, the functionality described in this disclosure may be provided as a service via a cloud environment. FIG. 10 is a schematic block diagram of a cloud-based system environment in which the functionality described herein may be provided as a cloud service, according to some embodiments. In the embodiment shown in FIG. 10, a cloud infrastructure system 1002 may provide one or more cloud services that may be requested by users using one or more client computing devices 1004, 1006, and 1008. The cloud infrastructure system 1002 may include one or more computers and / or servers, including those described above for server 912. The computers in the cloud infrastructure system 1002 may be organized as general-purpose computers, dedicated server computers, server farms, server clusters, or any other suitable arrangement and / or combination.

[0063] The network 1010 can facilitate communication and exchange of data between the clients 1004, 1006, and 1008 and the cloud infrastructure system. The network 1010 can include one or more networks. The networks can be of the same type or different types. The network 1010 can support one or more communication protocols, including wired and / or wireless protocols, to facilitate communication.

[0064] The embodiment shown in Figure 10 is merely one example of a cloud infrastructure system and is not intended to be limiting. It should be understood that in some embodiments, cloud infrastructure system 1002 may have more or fewer components than those shown in Figure 10, may combine two or more components, or may have a different configuration or arrangement of components. For example, while Figure 10 shows three client computing devices, alternative embodiments may support any number of client computing devices.

[0065] The term cloud service is generally used to refer to services made available to users on demand via a communication network, such as the Internet, by a service provider's system (e.g., cloud infrastructure system 1002). Typically, in a public cloud environment, the servers and systems that comprise the cloud service provider's system are separate from the customer's own on-premises servers and systems. The cloud service provider's systems are managed by the cloud service provider. This allows customers to use cloud services offered by the cloud service provider without having to separately purchase licenses, support, or hardware and software resources for the services. For example, the cloud service provider's system can host applications, and users can order and use the applications on demand via the Internet without having to purchase infrastructure resources to run the applications. Cloud services are designed to provide easy and scalable access to applications, resources, and services. Several providers offer cloud services. For example, several cloud services, such as middleware services, database services, and Java cloud services, are offered by Oracle Corporation®, located in Redwood Shores, California.

[0066] In one embodiment, cloud infrastructure system 1002 may provide one or more cloud services using various models, such as, for example, a Software as a Service (SaaS) model, a Platform as a Service (PaaS) model, an Infrastructure as a Service (IaaS) model, and other models, including hybrid service models. Cloud infrastructure system 1002 may include a set of applications, middleware, databases, and other resources that enable the provisioning of various cloud services.

[0067] The SaaS model allows applications or software to be delivered to customers as a service over a communications network such as the Internet, without the customer having to purchase the underlying application hardware or software. For example, the SaaS model can be used to provide customers with access to on-demand applications hosted by cloud infrastructure system 1002. Examples of SaaS services offered by Oracle Corporation® include, but are not limited to, various services for human resource / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytical services, social applications, etc.

[0068] The IaaS model is commonly used to provide infrastructure resources (e.g., servers, storage, hardware and networking resources) as cloud services to customers to provide elastic computing and storage capabilities. Various IaaS services are offered by Oracle Corporation.

[0069] The PaaS model is generally used to provide platform and environment resources that enable customers to develop, run, and manage applications and services as a service, without the customer having to procure, build, or maintain such resources. Examples of PaaS services provided by Oracle Corporation (registered trademark) include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), data management cloud services, and various application development solution services.

[0070] Cloud services are generally provided on an on-demand self-service basis, on a subscription basis, in an elastically scalable, reliable, highly available, and secure manner. For example, a customer may order one or more services provided by cloud infrastructure system 1002 via a subscription order. Cloud infrastructure system 1002 then performs processing to provide the services requested in the customer's subscription order. Cloud infrastructure system 1002 may be configured to provide one or even multiple cloud services.

[0071] Cloud infrastructure system 1002 can provide cloud services through different deployment models. In a public cloud model, cloud infrastructure system 1002 can be owned by a third-party cloud service provider, and cloud services are offered to public customers, which can be individuals or businesses. In certain other embodiments, under a private cloud model, cloud infrastructure system 1002 can operate within an organization (e.g., within a corporate organization) and services offered to customers within the organization. For example, customers can be various departments of a company, such as human resources, payroll, or even individuals within the company. In certain other embodiments, under a community cloud model, cloud infrastructure system 1002 and the services offered can be shared by multiple organizations within an associated community. Various other models, such as hybrids of the above models, can also be used.

[0072] Client computing devices 1004, 1006, and 1008 may be of different types (e.g., devices 902, 904, 906, and 908 shown in FIG. 9 ) and may be capable of running one or more client applications. Users may use the client devices to interact with cloud infrastructure system 1002, such as to request services provided by cloud infrastructure system 1002. For example, a user may use a client device to request data storage services described in this disclosure.

[0073] In some embodiments, the processing performed by cloud infrastructure system 1002 may include big data analysis. This analysis may include using, analyzing, and manipulating large data sets to detect and visualize various trends, behaviors, relationships, etc. within the data. This analysis may be performed by one or more processors, possibly processing the data in parallel, running simulations with the data, etc. The data used in this analysis may include structured data (e.g., data stored in a database or data structured according to a structured model) and / or unstructured data (e.g., data blobs (binary large objects)).

[0074] 10, cloud infrastructure system 1002 may include infrastructure resources 1030 utilized to facilitate provisioning of various cloud services provided by cloud infrastructure system 1002. Infrastructure resources 1030 may include, for example, processing resources, storage or memory resources, networking resources, etc.

[0075] In one embodiment, to facilitate efficient provisioning of these resources to support the various cloud services offered by cloud infrastructure system 1002 for the various customers, resources can be bundled into resource sets or resource modules (also called "pods"). Each resource module or pod can include a pre-integrated and optimized combination of one or more types of resources. In one embodiment, a different pod can be provisioned for database services, a second set of pods can be provisioned for Java services, and so on, which can include a different combination of resources than the pods in the first set of pods. For some services, resources allocated to provision the service can be shared between services.

[0076] Cloud infrastructure system 1002 may itself use services 1032 internally that are shared by different components of cloud infrastructure system 1002 and that facilitate provisioning of services by cloud infrastructure system 1002. These internal shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelisting services, high availability, backup and recovery services, services enabling cloud support, email services, notification services, file transfer services, etc.

[0077] Cloud infrastructure system 1002 may comprise multiple subsystems. These subsystems may be implemented in software or hardware, or a combination thereof. As shown in FIG. 10 , the subsystems may include a user interface subsystem 1012 that allows users or customers of cloud infrastructure system 1002 to interact with cloud infrastructure system 1002. User interface subsystem 1012 may include a variety of different interfaces, such as a web interface 1014, an online store interface 1016, and the like, through which cloud services offered by cloud infrastructure system 1002 are advertised and available for purchase by customers and other interfaces 1018. For example, a customer may access an online store, browse cloud services offered by cloud infrastructure system 1002, and place a subscription order for one or more services offered by cloud infrastructure system 1002 to which the customer wishes to subscribe. A service request may include information identifying the customer and the one or more services to which the customer wishes to subscribe.

[0078] 10, cloud infrastructure system 1002 may include an order management subsystem (OMS) 1020 configured to process new orders. As part of this processing, OMS 1020 may be configured to create an account for the customer if one does not already exist, receive billing and / or accounting information from the customer that will be used to bill the customer for providing the requested services to the customer, verify the customer information, and, once verified, book the customer's order and coordinate various workflows to prepare the order for provisioning.

[0079] Upon proper validation, the OMS 1020 can then invoke an order provisioning subsystem (OPS) 1024 configured to provision resources for the order, including processing, memory, and networking resources. Provisioning can include allocating resources for the order and configuring the resources to facilitate the services requested by the customer's order. The manner in which resources are provisioned for the order and the type of resources provisioned can vary depending on the type of cloud service ordered by the customer. For example, according to one workflow, the OPS 1024 can be configured to determine the specific cloud service being requested and identify the number of pods that may be pre-configured for that specific cloud service. The number of pods allocated for the order can vary depending on the size / amount / level / scope of the service being requested. For example, the number of pods allocated can be determined based on the number of users supported by the service, the duration for which the service is requested, etc. The allocated pods can then be customized for the specific requesting customer to provide the requested service.

[0080] The cloud infrastructure system 1002 may send a response or notification 1044 to the requesting customer to indicate when the requested service is ready for use. In some cases, information (e.g., a link) may be sent to the customer that enables the customer to begin using and taking advantage of the requested service.

[0081] Cloud infrastructure system 1002 may provide services to multiple customers. For each customer, cloud infrastructure system 1002 is responsible for managing information related to one or more subscription orders received from the customer, maintaining customer data related to the orders, and providing the requested services to the customer. Cloud infrastructure system 1002 may also collect usage statistics regarding the customer's use of the subscribed services. For example, statistics may be collected regarding the amount of storage used, the amount of data transferred, the number of users, system uptime and downtime, etc. This usage information may be used to bill the customer. Billing may occur, for example, on a monthly cycle.

[0082] Cloud infrastructure system 1002 can provide services to multiple customers concurrently. Cloud infrastructure system 1002 can store information for these customers, possibly including proprietary information. In one embodiment, cloud infrastructure system 1002 includes an identity management subsystem (IMS) 1028 configured to manage customer information and provide separation of management information so that information associated with one customer is not accessible by another customer. IMS 1028 can be configured to provide various security-related services, such as identity services, such as information access management, authentication and authorization services, customer identity and role management services, and related functions.

[0083] 11 illustrates an exemplary computer system 1100 that can be used to implement certain embodiments. For example, in some embodiments, the computer system 1100 can be used to implement any of the various servers and computer systems described above. As shown in FIG. 11, the computer system 1100 includes various subsystems, including a processing subsystem 1104 that communicates with a number of other subsystems via a bus subsystem 1102. These other subsystems can include a processing accelerator 1106, an I / O subsystem 1108, a storage subsystem 1118, and a communications subsystem 1124. The storage subsystem 1118 can include non-transitory computer-readable storage media, including a storage medium 1122 and a system memory 1110.

[0084] Bus subsystem 1102 provides a mechanism that allows the various components and subsystems of computer system 1100 to communicate with each other as intended. While bus subsystem 1102 is shown schematically as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 1102 may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a local bus using any of a wide variety of bus architectures, etc. For example, such architectures may include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, and a Peripheral Component Interconnect (PCI) bus. These may be implemented as mezzanine buses manufactured in accordance with the IEEE P1386.1 standard, etc.

[0085] The processing subsystem 1104 controls the operation of the computer system 1100 and may include one or more processors, application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). The processors may be single-core or multi-core processors. The processing resources of the computer system 1100 may be organized into one or more processing units 1132, 1134, etc. The processing units may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some embodiments, the processing subsystem 1104 may include one or more dedicated coprocessors, such as a graphics processor, a digital signal processor (DSP), etc. In some embodiments, some or all of the processing units of the processing subsystem 1104 may be implemented using customized circuitry, such as an ASIC or FPGA.

[0086] In some embodiments, the processing units in processing subsystem 1104 can execute instructions stored in system memory 1110 or computer-readable storage medium 1122. In various embodiments, the processing units can execute a wide variety of program or code instructions and can maintain multiple simultaneously executing programs or processes. At any given time, some or all of the program code being executed may reside in system memory 1110 and / or computer-readable storage medium 1122, potentially including one or more storage devices. Through appropriate programming, processing subsystem 1104 can provide the various functions described above. In instances where computer system 1100 is running one or more virtual machines, one or more processing units can be assigned to each virtual machine.

[0087] In some embodiments, a processing accelerator 1106 may optionally be provided to accelerate the overall processing performed by the computer system 1100, to perform customized processing, or to offload portions of the processing performed by the processing subsystem 1104.

[0088] I / O subsystem 1108 can include devices and mechanisms for inputting information into computer system 1100 and / or outputting information from or through computer system 1100. In general, use of the term input device is intended to include all possible types of devices and mechanisms for inputting information into computer system 1100. User interface input devices can include, for example, keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dials, buttons, switches, keypads, voice input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices can also include motion sensing and / or gesture recognition devices, such as a Microsoft Kinect® motion sensor that allows a user to control and interact with the input device, a Microsoft Xbox® 360 game controller, or devices that provide an interface for receiving input using gestures and voice commands. The user interface input devices may also include eye gesture recognition devices, such as a Google Glass® blink detector, that detects a user's eye movements (e.g., blinking when taking a photo and / or selecting a menu) and translates the eye gestures as input to the input device (e.g., Google Glass®). Additionally, the user interface input devices may also include voice recognition sensing devices that allow a user to interact with a voice recognition system (e.g., Siri® navigator) through voice commands.

[0089] Other examples of user interface input devices include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, gamepads and graphics tablets, and audio / visual devices such as speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode reader 3D scanners, 3D printers, laser range finders, eye-tracking devices, etc. Additionally, user interface input devices can include medical imaging input devices such as, for example, computed tomography, magnetic resonance imaging, position emission tomography, and medical ultrasound devices. User interface input devices can also include audio input devices such as, for example, MIDI keyboards, digital musical instruments, etc.

[0090] In general, use of the term output device is intended to include all possible types of devices and mechanisms for outputting information from computer system 1100 to a user or to another computer. User interface output devices can include non-visual displays, such as a display subsystem, indicator lights, or audio output devices. Display subsystems can also be flat-panel devices, such as those using cathode ray tubes (CRTs), liquid crystal displays (LCDs), or plasma displays, projection devices, touchscreens, etc. For example, user interface output devices can include, but are not limited to, a wide variety of display devices that visually convey textual, graphical, or audio / visual information, such as monitors, printers, speakers, headphones, automobile navigation systems, plotters, audio output devices, and modems.

[0091] The storage subsystem 1118 provides a repository or data store for storing information and data used by the computer system 1100. The storage subsystem 1118 provides a tangible, non-transitory, computer-readable storage medium for storing the basic programming and data structures that provide the functionality of some embodiments. The storage subsystem 1118 can store software (e.g., programs, code modules, instructions) that, when executed by the processing subsystem 1104, provide the functionality described above. The software can be executed by one or more processing units of the processing subsystem 1104. The storage subsystem 1118 can also provide a repository for storing data used in accordance with the teachings of the present disclosure.

[0092] The storage subsystem 1118 can include one or more non-transitory memory devices, including volatile and non-volatile memory devices. As shown in FIG. 11, the storage subsystem 1118 includes a system memory 1110 and a computer-readable storage medium 1122. The system memory 1110 can include multiple memories, including a volatile main random access memory (RAM) for storing instructions and data during program execution and a non-volatile read-only memory (ROM) or flash memory in which fixed instructions are stored. In some embodiments, a basic input / output system (BIOS), containing the basic routines that help to transfer information between elements within the computer system 1100, such as during start-up, can typically be stored in ROM. The RAM typically contains data and / or program modules currently being operated on and executed by the processing subsystem 1104. In some embodiments, the system memory 1110 can include multiple different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), etc.

[0093] 11, system memory 1110 may load executing application programs 1112, which may include various applications such as a web browser, a middle-tier application, a relational database management system (RDBMS), program data 1114, and an operating system 1116. By way of example, operating system 1116 may include Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.), and / or various versions of mobile operating systems, such as iOS, Windows® Phone, Android® OS, BlackBerry® OS, Palm® OS operating systems, etc.

[0094] The computer-readable storage medium 1122 can store programming and data structures that provide some example functionality. The computer-readable medium 1122 can provide storage of computer-readable instructions, data structures, program modules, and other data for the computer system 1100. Software (programs, code modules, instructions) that, when executed by the processing subsystem 1104, provide the functionality described above can be stored in the storage subsystem 1118. By way of example, the computer-readable storage medium 1122 can include non-volatile memory such as a hard disk drive, a magnetic disk drive, a CD-ROM, a DVD, an optical disk drive such as a Blu-Ray® disk, or other optical media. The computer-readable storage medium 1122 can include, but is not limited to, a Zip® drive, a flash memory card, a Universal Serial Bus (USB) flash drive, a Secure Digital (SD) card, a DVD disk, a digital video tape, etc. The computer-readable storage medium 1122 may also include SSDs based on non-volatile memory such as flash memory solid-state devices (SSDs), enterprise flash drives, solid-state ROMs, solid-state RAM such as dynamic RAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and volatile memory-based SSDs such as hybrid SSDs that combine DRAM and flash memory-based SSDs.

[0095] In some embodiments, the storage subsystem 1118 may also include a computer-readable storage medium reader 1120 that may be further coupled to a computer-readable storage medium 1122. The reader 1120 may be configured to receive and read memory from a memory device such as a disk, flash drive, or the like.

[0096] In some embodiments, computer system 1100 may support virtualization techniques, including, but not limited to, virtualization of processing and memory resources. For example, computer system 1100 may provide support for running one or more virtual machines. In some embodiments, computer system 1100 may execute a program, such as a hypervisor, that facilitates configuration and management of virtual machines. Each virtual machine may be assigned memory, computing (e.g., processors, cores), I / O, and network resources. Each virtual machine typically runs independently from other virtual machines. A virtual machine typically runs its own operating system, which may be the same or different from the operating systems run by other virtual machines executed by computer system 1100. Thus, multiple operating systems can potentially be run simultaneously by computer system 1100.

[0097] The communications subsystem 1124 provides an interface to other computer systems and networks. The communications subsystem 1124 serves as an interface for sending and receiving data from the computer system 1100 to and from other systems. For example, the communications subsystem 1124 may enable the computer system 1100 to establish communications channels with one or more client devices over the Internet for sending and receiving information to and from the client devices.

[0098] The communications subsystem 1124 may support both wired and / or wireless communications protocols. For example, in an embodiment, the communications subsystem 1124 may include radio frequency (RF) transceiver components for accessing wireless voice and / or data networks (e.g., using cellular technology, advanced data network technologies such as 3G, 4G, or EDGE (Enhanced Data Rates for Global Evolution)), WiFi (IEEE 802.XX family of standards, or other mobile communications technologies, or any combination thereof), global positioning system (GPS) receiver components, and / or other components. In some embodiments, the communications subsystem 1124 may provide a wired network connection (e.g., Ethernet) in addition to or instead of a wireless interface.

[0099] The communications subsystem 1124 can send and receive data in a variety of formats. For example, in some embodiments, in addition to other formats, the communications subsystem 1124 can receive incoming communications in the form of structured and / or unstructured data feeds 1126, event streams 1128, event updates 1130, etc. For example, the communications subsystem 1124 can be configured to receive (or send) data feeds 1126 in real time from users of social media networks and / or other communications services, such as web feeds such as Twitter® feeds, Facebook® updates, rich site summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0100] In one embodiment, the communications subsystem 1124 may be configured to receive data in the form of a continuous data stream, which may include an event stream 1128 of real-time events and / or event updates 1130 that has no explicit end and may be continuous or unlimited in nature. Examples of applications that may generate continuous data include sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc.

[0101] The communications subsystem 1124 may also be configured to communicate data from the computer system 1100 to other computer systems or networks. The data may be communicated in various forms, such as structured and / or unstructured data feeds 1126, event streams 1128, event updates 1130, etc., to one or more databases that may be in communication with one or more streaming data source computers coupled to the computer system 1100.

[0102] Computer system 1100 can be one of a variety of types, including a handheld portable device (e.g., an iPhone® mobile phone, an iPad® computing tablet, a PDA), a wearable device (e.g., a Google Glass® head-mounted display), a personal computer, a workstation, a mainframe, a kiosk, a server rack, or any other data processing system. Due to the ever-changing nature of computers and networks, the description of computer system 1100 shown in FIG. 11 is intended only as a specific example. Many other configurations are possible, having more or fewer components than the system shown in FIG. 11 .

[0103] While specific examples have been described, various modifications, variations, alternative configurations, and equivalents are possible. The embodiments are not limited to operation in a particular data processing environment, but can freely operate in multiple data processing environments. Additionally, while certain embodiments have been described using a particular sequence of transactions and steps, this is not intended to be limiting. While some flowcharts describe operations as a sequential process, many of the operations can be performed in parallel or simultaneously. Additionally, the order of operations may be rearranged. Processes may have additional steps not included in the figures. Various features and aspects of the above-described embodiments may be used individually or in combination.

[0104] Additionally, while certain embodiments have been described using particular combinations of hardware and software, it should be recognized that other combinations of hardware and software are possible. Certain embodiments may be implemented exclusively in hardware, exclusively in software, or using a combination thereof. The various processes described herein may be implemented on the same processor or on different processors in any combination.

[0105] Where a device, system, component, or module is described as being configured to perform a certain operation or function, such configuration may be achieved, for example, by designing an electronic circuit to perform the operation, by programming a programmable electronic circuit (e.g., a microprocessor) to perform the operation, such as by executing computer instructions or code, or a processor or core that is programmed to execute code or instructions stored on a non-transitory memory medium, or any combination thereof. Processes may communicate using a variety of techniques, including, but not limited to, conventional techniques for inter-process communication; different parts of a process may use different technologies, or pairs of the same processes may use different technologies at different times.

[0106] Specific details are provided in this disclosure to provide a thorough understanding of the embodiments. However, examples may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques are shown without unnecessary detail to avoid obscuring the embodiments. This description provides exemplary embodiments only and is not intended to limit the scope, applicability, or configuration of other embodiments. Rather, the description of the foregoing embodiments provides an useful description for implementing various examples. Various changes can be made in the function and arrangement of elements.

[0107] Accordingly, the specification and drawings should be regarded in an illustrative rather than a restrictive sense. It will be apparent, however, that additions, subtractions, deletions, and other modifications and changes may be made without departing from the broader spirit and scope of the appended claims. Accordingly, while specific embodiments have been described, they are not intended to be limiting. Various modifications and equivalents are intended to be encompassed within the scope of the following claims.

Claims

1. receiving client data from a client system, the client data including information corresponding to a set of users and / or a set of user devices; Identifying a privacy protocol to be applied to the client data; identifying a particular subset of said client data as personally identifiable information (PII) data, said subset including a set of PII attributes; Dividing the PII attributes into a plurality of predetermined categories based on a format of a data field in the PII attributes; For each predetermined category of the plurality of predetermined categories: selecting a processing technique from a plurality of processing techniques based on the predetermined category and the privacy protection protocol; processing PII data corresponding to the predetermined category using the selected processing technique; combining the processed PII data corresponding to the plurality of predetermined categories with non-PII data to create processed client data; determining, based on previous input from the client system and / or based on the processed PII data, to add noise to at least a portion of the processed PII data; determining an amount of noise based on the privacy preserving protocol; adding the amount of noise to the at least a portion of the processed PII data to generate protected training data; and training a machine learning model using the protected training data.

2. Identifying the privacy protection protocol includes: determining one or more geographic regions corresponding to the client system; and retrieving one or more data protection constraints corresponding to the one or more geographic regions.

3. Identifying the privacy protection protocol includes:

2. The computer-implemented method of claim 1, comprising receiving raw input from the client system identifying specifications for the privacy-preserving protocol to be applied with respect to the client system.

4. Identifying the privacy protection protocol includes:

2. The computer-implemented method of claim 1, comprising receiving input from the client system specifying how data protection is prioritized relative to performance metrics of the machine learning model.

5. Identifying the privacy protection protocol includes:

2. The computer-implemented method of claim 1, comprising receiving input from the client system via an interface presenting a predicted data protection indicator and a predicted performance indicator of the machine learning model for each of a plurality of potential data protection policies, the plurality of potential data protection policies including the privacy protection protocol.

6. The computer-implemented method of any one of claims 1 to 5, wherein the plurality of predetermined categories comprises categorical text data, categorical numeric data, high cardinality text, continuous numeric data, or text.

7. The computer-implemented method of any one of claims 1 to 6, wherein the plurality of processing techniques comprises using another ML model, using a rule-based algorithm, or performing a pattern recognition technique.

8. 1. A system comprising: one or more data processors; a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform operations, the operations including: receiving client data from a client system, the client data including information corresponding to a set of users and / or a set of user devices; Identifying a privacy protocol to be applied to the client data; identifying a particular subset of said client data as personally identifiable information (PII) data, said subset including a set of PII attributes; Dividing the PII attributes into a plurality of predetermined categories based on a format of a data field in the PII attributes; For each predetermined category of the plurality of predetermined categories: selecting a processing technique from a plurality of processing techniques based on the predetermined category and the privacy protection protocol; processing PII data corresponding to the predetermined category using the selected processing technique; combining the processed PII data corresponding to the plurality of predetermined categories with non-PII data to create processed client data; determining, based on previous input from the client system and / or based on the processed PII data, to add noise to at least a portion of the processed PII data; determining an amount of noise based on the privacy preserving protocol; adding the amount of noise to the at least a portion of the processed PII data to generate protected training data; and training a machine learning model using the protected training data.

9. Identifying the privacy protection protocol includes: determining one or more geographic regions corresponding to the client system; and retrieving one or more data protection constraints corresponding to the one or more geographic regions.

10. Identifying the privacy protection protocol includes:

10. The system of claim 8, further comprising receiving input directly from the client system identifying specifications for the privacy protection protocol to be applied with respect to the client system.

11. Identifying the privacy protection protocol includes:

10. The system of claim 8, further comprising receiving input from the client system specifying how data protection is prioritized relative to the performance metrics of the machine learning model.

12. Identifying the privacy protection protocol includes:

10. The system of claim 8, further comprising receiving input from the client system via an interface presenting a predicted data protection indicator and a predicted performance indicator of the machine learning model for each of a plurality of potential data protection policies, the plurality of potential data protection policies including the privacy protection protocol.

13. The system of any one of claims 8 to 12, wherein the plurality of predetermined categories comprises categorical text data, categorical numeric data, high cardinality text, continuous numeric data, or text.

14. The system of any one of claims 8 to 13, wherein the plurality of processing techniques includes using another ML model, using a rule-based algorithm, or performing a pattern recognition technique.

15. A computer program product tangibly embodied in a non-transitory computer-readable storage medium, the computer program product comprising instructions configured to cause one or more data processors to perform operations, the operations including: receiving client data from a client system, the client data including information corresponding to a set of users and / or a set of user devices; Identifying a privacy protocol to be applied to the client data; identifying a particular subset of said client data as personally identifiable information (PII) data, said subset including a set of PII attributes; Dividing the PII attributes into a plurality of predetermined categories based on a format of a data field in the PII attributes; For each predetermined category of the plurality of predetermined categories: selecting a processing technique from a plurality of processing techniques based on the predetermined category and the privacy protection protocol; processing PII data corresponding to the predetermined category using the selected processing technique; combining the processed PII data corresponding to the plurality of predetermined categories with non-PII data to create processed client data; determining, based on previous input from the client system and / or based on the processed PII data, to add noise to at least a portion of the processed PII data; determining an amount of noise based on the privacy preserving protocol; adding the amount of noise to the at least a portion of the processed PII data to generate protected training data; training a machine learning model using the protected training data; a computer program product,

Images