Privacy enhanced machine learning

Abstract

A method of selecting data for privacy-preserving machine learning includes storing training data from a first participant, storing a machine learning model, and storing criteria from the first participant or another participant. The method includes filtering training data to select a first portion of the training data to be used to train a machine learning model and to select a second portion of the training data. Selection is accomplished by using a measure of the contribution of standard computational data to machine learning model performance.

Description

[0001] This application is a divisional application of the Chinese invention patent application with an application date of June 18, 2020, Chinese national application number 202080065917.3, and invention name “Privacy Enhanced Machine Learning”. Background Art

[0002] In machine learning, and particularly deep learning, a major obstacle to training well-performing machine learning models is often the limited availability of sufficiently diverse, labeled training data. While the data needed to train good models often exists, it's difficult to leverage because it's distributed and owned by multiple parties. For example, in the healthcare field, crucial patient data that could be used to learn a cancer diagnosis support system may be held by different hospitals, each of which maintains different data from specific geographic regions with varying demographics.

[0003] The embodiments described below are not limited to implementations that solve any or all shortcomings of known machine learning systems. Summary of the Invention

[0004] The following presents a simplified summary of the present disclosure to provide the reader with a basic understanding. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Its sole purpose is to present a selection of concepts disclosed herein in a simplified form as a prelude to the more detailed description that is presented later.

[0005] In various examples, a method for selecting data for privacy-preserving machine learning includes storing training data from a first party, storing a machine learning model, and storing criteria from the first party or another party. The method includes filtering the training data to select a first portion of the training data to be used to train the machine learning model and selecting a second portion of the training data. The selection is performed by calculating a measure of the contribution of the data to the performance of the machine learning model using the criteria.

[0006] Many of the attendant features will be more readily appreciated as they become better understood by reference to the following detailed description considered in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] This description will be better understood through the following detailed description read in conjunction with the accompanying drawings, in which

[0008] Figure 1 A schematic diagram of a multi-party machine learning system used by two hospitals;

[0009] Figure 2 yes Figure 1A more detailed diagram of a multi-party machine learning system;

[0010] Figure 3 is a schematic diagram of another multi-party machine learning system;

[0011] Figure 4 is a flowchart of a method for privacy-preserving training performed by a multi-party machine learning system, wherein a single machine learning model is being trained;

[0012] Figure 5 is a flowchart of a method for privacy-preserving training performed by a multi-party machine learning system, wherein multiple machine learning models are being trained;

[0013] Figure 6 is a flow chart of a method for controlling access to a single trained machine learning model;

[0014] Figure 7 is a flow chart of a method of controlling access to a plurality of trained machine learning models;

[0015] Figure 8 Illustrated are exemplary computing-based devices in which embodiments of a multi-party machine learning system are implemented.

[0016] In the drawings, the same reference numerals are used to denote the same components. DETAILED DESCRIPTION

[0017] The detailed description provided below in conjunction with the accompanying drawings is intended as a description of this example and is not intended to represent the only way to construct or utilize this example. This description sets forth the functions of the example and the order of operations for constructing and operating the example. However, the same or equivalent functions and orders can be implemented by different examples.

[0018] As mentioned above, in the medical field, important patient data that can be used to learn cancer diagnosis support systems may be held by different hospitals, each of which maintains different data from specific geographic regions with different demographic characteristics. By pooling available data, hospitals can train better machine learning models for their applications, rather than using only their own data. Since all hospitals benefit from better machine learning models obtained through data sharing, collaborative machine learning is necessary.

[0019] Of course, this type of collaboration raises technical challenges related to one or more of the following: the security of the individual participants' data, the privacy of the individual participants' data, and the quality of the machine learning results. It is difficult to deploy a collaborative machine learning system that respects the integrity of the individual participants' data. Integrity, in this context, involves ensuring that a participant's training data is not modified during collaborative training and that the data submitted by the participant is actually used for collaborative training.

[0020] although Figure 1 Described and illustrated herein as being implemented for use by a hospital, the system described is provided as an example rather than a limitation. Those skilled in the art will appreciate that this example is applicable to applications in many different types of multi-party machine learning scenarios.

[0021] Figure 1 is a schematic diagram of a multi-party machine learning system 100 being used by two hospitals 108 and 112. The multi-party machine learning system 100 is deployed in a trusted execution environment 104 in the cloud or at any location that is in communication with the hospital via a communication network 106 such as the Internet, an intranet, or any other communication network 106. The trusted execution environment 104 is implemented using secure hardware and / or software and includes at least one secure memory area. In this example, for clarity, one trusted execution environment 104 is shown, although in practice many trusted execution environments 104 can be deployed and located at computing units in a data center, such as servers with disk storage or virtual machines connected via a network within the data center. Where there are many trusted execution environments 104, the multi-party machine learning system 100 is distributed among them.

[0022] In one example, the trusted execution environment 104 includes a secure memory area, which is a processor-protected memory area within the address space of a regular process. The processor monitors memory access to the trusted execution environment so that only code running in the trusted execution environment 104 can access data in the trusted execution environment 104. When inside the physical processor package (in the processor's cache), the memory of the trusted execution environment 104 is available in plain text, but when written to system memory (random access memory RAM), it is encrypted and integrity protected. External code can only call code inside the trusted execution environment at statically defined entry points (using a call-gate mechanism).

[0023] In some examples, the trusted execution environment 104 is implemented using hardware so that the secure memory area is isolated from any other code including the operating system and hypervisor. In some examples, the trusted execution environment 104 is implemented using a trusted virtual machine.

[0024] Within the trusted execution environment 104 are one or more trained machine learning models 102 that have been computed by the multi-party machine learning system 100 using training data from multiple parties, such as Figure 1 In the example shown in Figure 1, there is a first hospital 108 and a second hospital 112. First hospital 108 has training data 110, such as medical images of patients, where the medical images are labeled as depicting or not depicting cancer. Training data 110 is confidential and securely stored at first hospital. When the hospitals want to participate in multi-party machine learning, training data 110 is encrypted and uploaded to trusted execution environment 104.

[0025] Second hospital 112 has training data 114, which includes medical images of different patients, where the images are labeled as depicting or not depicting cancer. Training data 114 is confidential and securely stored at second hospital. When second hospital wants to participate in multi-party machine learning, training data 114 is encrypted and uploaded to trusted execution environment 104.

[0026] One or more malicious parties, such as malicious party 116, may exist and have fake training data 118. Fake training data is any training data of poor quality, such as having inaccurate labels, or a duplication of training data that has been uploaded to the trusted execution environment by the party.

[0027] One or more parties upload training data to the trusted execution environment 104. The multi-party machine learning system 100 uses at least some of the training data to train one or more machine learning models 102. One or more of the parties can then access the trained machine learning model and use it to compute predictions to label medical images for tumor detection or other tasks, depending on the application domain. In this way, a first party, such as hospital one, can benefit from a high-performance machine learning model that has been trained using data from multiple parties. The performance of the resulting machine learning model would be lower if the first party used only its own training data, where the amount and / or variety of data is typically lower than the amount and / or variety of data available to the multiple parties.

[0028] Figure 1 The multi-party machine learning system allows multiple participants to jointly train machine learning models based on the training data provided by all participants, achieving improved performance on their respective tasks. The multi-party machine learning system supports scenarios with a single validation task, such as hospitals pooling their data to train a single model for cancer detection. Furthermore, the multi-party machine learning system supports scenarios where a single participant's data contributes to multiple tasks.

[0029] Assume that training data from the first hospital 110 and the second hospital 112 are uploaded to the trusted execution environment 104 and used by the multi-party machine learning system to train one or more machine learning models in the machine learning model 102. Then, assume that the first hospital and the second hospital can access the resulting trained machine learning model 102 via the communication network 106. The first hospital may then discover information about the training data used to train the machine learning model 102. Therefore, the first hospital is able to discover the confidential training data of the second hospital. Attacks that obtain confidential training data from prediction application programming interfaces are known, such as those described by Tramer et al. in "Stealing machine learning models via prediction APIs" in USENIX Security 2016.

[0030] Various examples described herein use a selector component within a multi-party machine learning system to enhance privacy. The selector component selects a first portion of training data from training data uploaded by multiple parties and uses only the first portion of the training data to train one or more machine learning models. A second portion of the training data remains private within the trusted execution environment. The selection is done based on one or more criteria submitted by the individual parties. In this way, at least some of the training data that has been uploaded to the trusted execution environment 104 is not used to train a particular machine learning model instance. As a result, privacy is enhanced because a malicious party with access to a trained machine learning model cannot discover unused training data. Although some, but not all, training data is used, the performance of the machine learning model is not affected by carefully designing the selection process. In some cases, the criteria include validation data, and the use of a selector provides the following benefits: only information relevant to the validation task of the validation data is published by the model, thereby limiting the possibility of copying the training data and reusing it for other tasks.

[0031] Various examples described herein use a participant controller within a multi-party machine learning system to improve the quality of the resulting trained machine learning model 102 and prevent errors such as Figure 1 A spoofing attack in which a malicious participant 116 uses fake training data (such as already used training data) to gain access to the trained machine learning model 102. The participant controller calculates a measure of an individual participant's contribution to a particular trained machine learning model and uses the measure to control access to that model or other machine learning models.

[0032] Figure 2 Such as Figure 1A more detailed diagram of a multi-party machine learning system of a multi-party machine learning system. The trusted execution environment 104 includes the multi-party machine learning system 100.

[0033] The multi-party machine learning system 100 includes a memory that stores training data 200 and a model library 202 that stores at least one machine learning model. The multi-party machine learning system 100 optionally includes a selector 204, and includes a standard storage device 206, a training engine 208, a participant controller 210, and a storage device for holding one or more trained machine learning models 102 calculated by the training engine 208.

[0034] Because it is stored within the trusted execution environment, the stored training data 200 is stored in plain text. The training data 200 includes a plurality of examples, such as images, videos, documents, sensor data values, or other training examples. The training data 200 includes labeled training data when the training engine 208 uses supervised training and / or unlabeled training data when unsupervised training is used. The stored training data 200 has been received from two or more participants at the trusted execution environment 104. Figure 2 In FIG, two participants are shown as 108 and 112, but there may be more participants in practice. When a participant uploads training data, the training data is encrypted. The training data 200 stored in the trusted execution environment is tagged or marked to indicate which participant it originated from.

[0035] The model library 202 is a storage device for one or more machine learning models, such as a neural network, a random decision forest, a support vector machine, a classifier, a regressor, or other machine learning models.

[0036] The selector 204 is optional and is included in situations where a multi-party machine learning system enhances privacy by selecting some, but not all, of the training data 200 to be used to train a particular instance of a machine learning model. The selector uses one or more criteria provided by each individual one of the parties. Figure 2 It shows that participant 1 uploads standards 220 and training data 222 to the trusted execution environment. It also shows that participant 2 uploads standards 224 and training data 226 to the trusted execution environment.

[0037] The criteria storage 206 holds the criteria uploaded by each of the participants. A criterion is a quality, threshold, value, metric, statistic, or other standard used to select training data and / or indicate the performance level of a machine learning model.

[0038] The training engine 208 is one or more training processes for training machine learning models from the model library 202. In some examples, the training process is a well-known conventional training process.

[0039] Participation controller 210 includes functionality for calculating a measure of the contribution of individual participants' training data to the performance of a particular trained machine learning model. In some examples, the participation controller uses a standard. More details about the participation controller are provided later in this document.

[0040] Trained machine learning model 102 is the stored architecture, parameter values, and other data that specifies a single trained machine learning model.

[0041] The access controller 212 is a firewall, network card, or other functionality that enables control of access to the trained machine learning model 102 by individual parties 108 , 112 .

[0042] The selector of a multi-party machine learning system operates in an unconventional manner to enhance the privacy of a trained machine learning model without compromising the performance of the machine learning model.

[0043] A selector of a multi-party machine learning system improves functionality of an underlying computing device by selecting a first portion of training data to be used to train the machine learning model and selecting a second portion of the training data to remain private, thereby maintaining performance of a trained machine learning model.

[0044] Participating controllers of a multi-party machine learning system operate in an unconventional manner to protect access to trained machine learning models.

[0045] A participant controller of a multi-party machine learning system improves the functionality of underlying computing devices by increasing the security of access to trained machine learning models and preventing spoofing attacks in which malicious participants spoof training data in an attempt to gain access to the trained machine learning models.

[0046] Alternatively or additionally, reference Figure 2 The functions described herein are performed at least in part by one or more hardware logic components. For example, but not limited to, illustrative types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and graphics processing units (GPUs).

[0047] Figure 3 is with Figure 2Schematic diagram of a multi-party machine learning system that is very similar and in which like components have like reference numerals. In this example, the criteria uploaded by each party are tasks, where the tasks are verification data for verifying the machine learning tasks. Figure 3 As shown, participant 1 108 uploads the encrypted training data to the training data server 200 in the trusted execution environment and uploads the task (in this example, the task is verification data) to the standard storage device 206 (in Figure 3 There are multiple participants, each of which can participate in multi-participant machine learning only if it uploads a standard such as a task (i.e., verification data in this example) to the multi-participant machine learning system. Participant M 112 uploads encrypted training data to the training data server 200 in the trusted execution environment and uploads the task (in this example, the task is verification data) to the standard storage device 206 (in Figure 3 is referred to as the task server). Figure 3 The padlocks 300, 302 in the figure represent access control mechanisms, such as firewalls, network cards, software, or other access controls that prevent each participant from accessing the multi-participant machine learning system unless it has uploaded standards, such as verification data in this example.

[0048] Using the uploaded training data, the multi-party machine learning system performs machine learning. In some examples, it trains a single machine learning model, which each of the participants may then access. In some examples, it trains multiple machine learning models, one for each of the participants.

[0049] In some examples, the use is Figure 2 The selector 204 is a task and data matching component of the selector 204. The selector uses the uploaded criteria to select which training data items are used to train specific ones of the models and which training data items remain private and are not used to train specific ones of the models. As mentioned above, the criteria are qualities, thresholds, values, metrics, statistics, or other criteria used to select training data and / or indicate the performance level of the machine learning model. In one example, the criteria include validation data, and in this case, the validation data is used to evaluate the performance of the trained machine learning model in a conventional manner. In one example, the criteria include qualities such as roundness, where it is known that for a specific result, images depicting round objects will be better for training the machine learning system. In one example, the criteria include a number of corrupted bits, where it is known that for a specific result, an audio signal with a high level of corrupted bits will be worse for training the machine learning system.

[0050] Customized machine learning model training components and Figure 2The training engine 208 is the same as the training engine 208. It uses the selected training data to train the machine learning model from the machine learning model library 202. The output of the training engine 208 is stored in the trained machine learning model 102.

[0051] The stored trained machine learning model 102 is Figure 2 The access controller 212 Figure 3 The engagement controller 210 calculates scores that indicate whether (and optionally for how long) each individual participant has access to each individual trained machine learning model 102 in the trained machine learning model 102.

[0052] Since the number of participants changes over time as different participants leave or join the multi-participant machine learning system, Figure 3 The apparatus of FIG. 2 is shown for a particular instance in time. Thus, the training data at storage device 200, the criteria at storage device 206, and the trained machine learning models at storage device 102 change over time as the apparatus operates. It is possible that, depending on the calculations of participant controller 210, a participant may initially access individual ones of the machine learning models but lose that access over time (and potentially regain access again). A participant can participate in the multi-participant machine learning system as long as it submits criteria.

[0053] Figure 4 4 is a flow chart of a method for privacy-preserving training performed by a multi-party machine learning system, wherein a single machine learning model is being trained. The machine learning model is stored 400, such as by selecting the machine learning model from a model library 202 according to one or more rules or according to a selection parameter value given by one of the participants. Figure 2 The first participant of participant 1 receives 402 training data. Figure 2 One or more participants of participant 2 receive 404 training data.

[0054] The multi-party machine learning system 100 checks 406 whether criteria have been received from the first party. If not, the multi-party machine learning system waits and continues to check for the arrival of criteria. If criteria have been received from party one, the process continues by using a selector 204. The selector selects a first portion of training data to be used to train the stored machine learning model. The selector selects a second portion of training data to be kept private and not used to train the stored machine learning model. The selection is done based on the criteria from party one.

[0055] The multi-party machine learning system uses the first portion of the training data to train 410 the stored machine learning model. The selection of the criterion to use is done in a manner that does not harm the performance of the resulting trained model compared to the performance of the model trained using all available training data.

[0056] In some examples, the resulting trained machine learning model is deployed 412 by retaining it in a trusted execution environment and allowing access to the trained machine learning model via the access controller 212. Parties that send queries to and obtain results from the trained machine learning model in the trusted execution environment cannot reverse engineer training data that remains private and was not used to train the machine learning model. In some cases, the access controller 212 and the scores from the participation controller 208 are used to control 414 the ability of parties to send queries to and obtain results from the trained machine learning model, as described in more detail later in this document.

[0057] In some examples, the resulting trained machine learning model is deployed 412 by installing it on an end-user device or on a server outside the trusted execution environment. In this case, security is enhanced compared to deploying the trained machine learning model after training it on all available training data. A malicious party that attacks the deployed machine learning model to obtain the used training data cannot obtain the training data that remains private in the trusted execution environment.

[0058] Figure 5 is a flowchart of a method for privacy-preserving training performed by a multi-party machine learning system, wherein multiple machine learning models are being trained. Figure 3 An example is given in which each party has a machine learning model.

[0059] The multi-party machine learning system receives 500 training data from a first party and receives 502 training data from one or more other parties. The multi-party machine learning system checks 504 whether it has received standards from each party. Each party that has submitted standards can participate. Figure 3 Access controls 300, 302 prevent an entity that has not submitted standards from accessing the multi-party machine learning system and is therefore not a participant.

[0060] For each participant, the multi-party machine learning system selects 506 some training data, but not all training data, based on the corresponding participant's criteria. For each participant, the multi-party machine learning system trains 508 a machine learning model using the appropriate selected training data.

[0061] After the individual machine learning models have been trained, they are deployed 510. Deploying the individual models is accomplished by enabling access controller 212 to allow participants associated with the individual models to send queries and receive responses from the individual models. In some cases, access is controlled 512 based on a score calculated by engagement controller 210, as described below. However, use of engagement controller 210 is not required.

[0062] Figure 6 6 is a flow chart of a method for controlling access to a single trained machine learning model that has been trained by a multi-party machine learning system. The multi-party machine learning system stores 600 machine learning models. The machine learning model is selected from a model library 202 based on one or more rules or based on a selection parameter value given by one of the participants. Figure 2 A first participant of participant one receives 602 training data and receives training data from a participant such as Figure 2 One or more participants of participant two receive 604 training data.

[0063] The multi-party machine learning system 100 checks whether the criteria have been received from the first party 606. If not, the multi-party machine learning system waits and continues to check for the arrival of the criteria.

[0064] The multi-party machine learning system uses some or all of the training data to train 608 the stored machine learning model.

[0065] For each participant, the participant controller calculates 610 a measure of the contribution of the training data submitted by the participant to the performance of the trained machine learning model. The measure of the contribution is calculated using the criteria submitted by participant 1.

[0066] The resulting trained machine learning model is deployed 412 by retaining it in a trusted execution environment and allowing access to the trained machine learning model via the access controller 212. The access granted to a participant is related to a measure of contribution calculated for that participant. For each participant, a check 612 is performed to see if the measure of contribution is above a threshold. If so, access to the trained model is granted 616. If not, access is blocked at 614.

[0067] In some cases, Figure 4 and Figure 6That is, the machine learning model uses the Figure 4 The selected training data is selected for training according to the description of Figure 6 Controls access to trained models.

[0068] Figure 7 is a flow diagram of a method performed by a multi-party machine learning system in which multiple machine learning models are being trained and in which access to the individually trained models is controlled using a participation controller 210 and an access controller 212.

[0069] The multi-party machine learning system receives 700 training data from a first party and receives 702 training data from one or more other parties. The multi-party machine learning system checks 704 whether it has received standards from each party. Each party that submitted a standard can participate. Figure 3 Access controls 300, 302 prevent an entity that has not submitted a standard from accessing the multi-party machine learning system and is therefore not a participant.

[0070] For each participant, the multi-participant machine learning system trains 706 a machine learning model using all or some of the training data (thus, it may be trained using all of the training data submitted by all participants).

[0071] For each participant, the multi-participant machine learning system calculates 708 a measure of the contribution of the participant's training data to the performance of each of the machine learning models.

[0072] For each participant and each model, the multi-participant machine learning system checks 712 to see if the contribution metric is above a threshold. If so, the particular participant is granted access to the trained model 716. If not, access is blocked at 714.

[0073] In various examples, the selector 204 and the participant controller 210 calculate Shapley values. The Shapley value is the output of a function that takes a characteristic function and a participant i as parameters. The characteristic function used by the selector 204 is different from the characteristic function used by the participant controller 210.

[0074] The characteristic function v and the Shapley value of the participant i∈M are as follows:

[0075]

[0076] Expressed in words as follows: the Shapley value of a feature function v and a participant i that is a member of a set M of participants of a multi-participant machine learning system is given by the sum of the factorial of the cardinality of the set S over every possible set S of M participants that does not include i, multiplied by the factorial of the number of participants M minus the cardinality of the set S minus 1, divided by the factorial of the number of participants M, multiplied by the difference in the output of the feature function of S with i and S without i.

[0077] The Shapley value quantifies the average marginal contribution of participant i to a subset of all possible participants. The inventors have recognized that Shapley values are not robust to replication, that is, they do not take into account participants that submit the same training data multiple times.

[0078] The selector 204 uses the following characteristic functions when calculating the Shapley value for the case of a single machine learning model and the case of one machine learning model per participant:

[0079]

[0080] In words: the characteristic function used by the selector 204 when calculating the Shapley value is considered as a parameter of the number of possible sets S of participants and is equal to the gain function The output of the machine learning model M has been trained using the data available from all the combinations of parties in one of the sets j in the possible set S. After training, when using the standard (such as verification data provided by participant i) for this specific machine learning model When evaluating, the gain function expresses the specific machine learning model The performance of To express the characteristic function, so as to express the gain function Use as a characteristic function.

[0081] In the case of a single machine learning model, the participating controller 210 uses the following feature function when calculating the Shapley value:

[0082]

[0083] Expressed in words, the characteristic function used by the participating controller 210 for S possible sets of participants in a multi-participant machine learning system is equal to the sum of the performance of the model plus the performance of each individual participant's model. The symbol v is used to refer to the characteristic function used by the participating controller for a single machine learning model.

[0084] The characteristic function is the value of the model trained on all datasets in S plus the marginal benefit of each participant. Note that for a single participant, the value of the data is expressed as the value of the model trained on its own training dataset.

[0085] In the case of one machine learning model per participant, the participant controller 210 uses the following feature function when calculating the Shapley value:

[0086]

[0087] Expressed in words, the characteristic function used by participating controller 210 when calculating the Shapley value in the case of one machine learning model per participant is equal to the sum of the performance of all models plus the sum of the performance gains of each model for each individual participant, based only on the data of the individual participants. The symbol ω is used to refer to the characteristic function used by the participating controller in the case of multiple machine learning models.

[0088] Figure 8 Various components of an exemplary computing-based device 800 are illustrated, which may be implemented as any form of computing device and / or electronic device and in which, in some examples, embodiments of a multi-party machine learning system are implemented.

[0089] The computing-based device 800 includes one or more processors 802, which are microprocessors, controllers, or any other suitable type of processors, for processing computer-executable instructions to control the operation of the device to train one or more machine learning models using training data from one or more participants. In some examples, such as where a system-on-chip architecture is used, the processor 802 includes one or more fixed function blocks (also referred to as accelerators) that are implemented in hardware (rather than software or firmware). Figures 4 to 7 806. A trusted execution environment 104 is provided to maintain training data and machine learning models, as previously described herein.

[0090] Computer-executable instructions are provided using any computer-readable medium accessible to the computing-based device 800. Computer-readable media include, for example, computer storage media such as memory 808 and communication media. Computer storage media such as memory 808 include volatile and non-volatile, removable, and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, and the like. Computer storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electronically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium for storing information for access by a computing device. In contrast, communication media embodies computer-readable instructions, data structures, program modules, and the like in a modulated data signal such as a carrier wave or other transmission mechanism. As defined herein, computer storage media does not include communication media. Therefore, computer storage media themselves should not be construed as propagating signals. Although computer storage media (memory 808 ) is shown within computing-based device 800 , it should be appreciated that in some examples storage is distributed or remotely located and accessed via a network or other communication link (eg, using communication interface 810 ).

[0091] The computing-based device 800 also includes an input / output controller 812 that is arranged to output display information to a display device 814, which may be separate from or integrated into the computing-based device 800. The display information may provide a graphical user interface. The input / output controller 812 is also arranged to receive and process input from one or more devices such as a user input device 816 (e.g., a mouse, keyboard, camera, microphone, or other sensor).

[0092] Alternatively or in addition to other examples described herein, examples include any combination of:

[0093] A method for selecting data for privacy-preserving machine learning, comprising:

[0094] storing training data from the first party;

[0095] Storing machine learning models;

[0096] storing criteria from the first party or another party;

[0097] select training data to select a first portion of the training data to be used to train the machine learning model and select a second portion of the training data;

[0098] The selection is done by using a standard measure of the contribution of the data to the performance of the machine learning model.

[0099] This enhances privacy because the first portion of the training data can be used to train the machine learning model without using the second portion. Therefore, the second portion cannot be revealed by accessing the trained model. By using the criteria for selection, the performance of the model is improved even if the model was not trained on all available training data.

[0100] The method described above is performed in a trusted execution environment and includes training a machine learning model using a first portion of training data, such that a second portion of the training data remains private in the trusted execution environment. Security is enhanced by using a trusted execution environment.

[0101] The method described above, wherein the criteria include one or more of: quality, threshold, value, metric, statistic. Since the trusted computing environment is a resource-constrained entity, these calculations and storage in the multi-party machine learning system are efficient.

[0102] The method described above wherein the criteria are adapted to select training data based on the likelihood of the machine learning model's performance when trained using the selected training data. This improves the performance of the machine learning model even if not all available training data is used. Efficiency is gained.

[0103] The method described above, where a criterion is applied to indicate the performance level of a machine learning model.

[0104] The method described above, wherein the standard is validation data used to validate the machine learning task for which the machine learning model is to be trained.

[0105] The method described above, in which the metric is a Shapley value calculated using a characteristic function, where the characteristic function is equal to the performance of the machine learning model when evaluated using a criterion given by participant i after the model has been trained using training data available from all combinations of participants in one of a plurality of possible sets S of participants, provides an efficient and practical way to select training data that has been found to perform well in empirical testing.

[0106] The method described above, wherein there are multiple machine learning models. Using multiple machine learning models provides flexibility and enables different parties to train different models.

[0107] The method described above includes calculating a metric of the contribution of a first party's training data to the performance of a machine learning model, and controlling access to the machine learning model based on the calculated metric. In this way, a malicious party that submits duplicate training data and / or poor quality training data is prevented from accessing the results.

[0108] The method described above, where the contribution metric is a Shapley value calculated using a characteristic function, where the characteristic function is equal to the sum of the performance of the machine learning model plus the performance of the machine learning model for each individual participant, was found to perform well in empirical testing.

[0109] The method described above includes storing multiple machine learning models, one for each participant, and wherein the metric is a Shapley value calculated using a characteristic function, where the characteristic function is equal to the sum of the performance of all machine learning models plus the sum of the performance of each machine learning model for each individual participant. The characteristic function used in this article has been found to perform well in practice.

[0110] An apparatus for selecting data for privacy-preserving machine learning, comprising:

[0111] a memory for storing training data from the first party;

[0112] Memory, which stores machine learning models;

[0113] a memory storing criteria from the first party or another party;

[0114] A selector configured to select training data to select a first portion of the training data to be used to train the machine learning model and to select a second portion of the training data; wherein the selecting is performed by calculating a measure of the contribution of the data to the performance of the machine learning model using a standard.

[0115] An apparatus for controlling access to a machine learning model, the apparatus comprising:

[0116] Trusted computing environment, which stores machine learning models and training data;

[0117] Access controllers, which are configured to allow or deny access to machine learning models;

[0118] a storage to store criteria submitted by parties requesting access to the machine learning model;

[0119] Participating controllers, using standard calculation scores;

[0120] And wherein the access controller uses the calculated score to allow or deny access to the machine learning model.

[0121] The apparatus described above, wherein the criterion is suitable for indicating the performance of a machine learning model.

[0122] The apparatus described above, wherein the access controller is configured to prevent parties who submit training data to the trusted computing environment rather than submitting standards to the trusted computing environment from accessing the machine learning model.

[0123] The apparatus described above, wherein the access controller uses the calculated score to grant timed access to the machine learning model, the timing being related to the score.

[0124] The apparatus described above, wherein training data has been submitted by one or more parties, and wherein the access controller prevents access to the machine learning model by a malicious party that submitted the submitted training data.

[0125] The apparatus described above, wherein the participating controller calculates a score as a Shapley value using a characteristic function, wherein the characteristic function is equal to the sum of the performance of the machine learning model plus the performance of the machine learning model for each individual participant.

[0126] The apparatus described above, wherein the trusted computing environment stores multiple machine learning models, one machine learning model for each participant, and the participating controller calculates a score using a characteristic function as a Shapley value, wherein the characteristic function is equal to the sum of the performance of all machine learning models plus the sum of the performance of each machine learning model for each individual participant.

[0127] A method for controlling access to a machine learning model, the method comprising:

[0128] Storing machine learning models and training data in a trusted computing environment;

[0129] Use access controllers to allow or deny access to machine learning models;

[0130] storing, at a memory, criteria submitted by a party requesting access to a machine learning model;

[0131] Scores were calculated using criteria;

[0132] And use the calculated score to allow or deny access to the machine learning model.

[0133] The disclosure presented herein includes the subject matter described in the following clauses.

[0134] Clause 1. A system comprising: a processor; and a memory storing instructions that, when executed by the processor, perform operations comprising: executing a trusted computing environment configured to store a machine learning model and training data; storing criteria for accessing the machine learning model, the criteria indicating the performance of the machine learning model; calculating a score using the criteria and a characteristic function, the characteristic function being equal to the performance of the machine learning model plus the sum of the performance of the machine learning model for each individual participant; and controlling access to the machine learning model based on the calculated score.

[0135] Clause 2. A system according to clause 1, wherein the criteria includes verification data, based on which only information relevant to the verification task is released through the machine learning model.

[0136] Clause 3. A system according to clause 1, wherein the processor performs additional operations, the additional operations comprising: preventing another participant who submits the training data to the trusted computing environment instead of submitting the standard from accessing the machine learning model.

[0137] Clause 4. A system according to clause 1, wherein the processor performs additional operations, the additional operations comprising: using the calculated score to grant timed access to the machine learning model, the timing being related to the score.

[0138] Clause 5. A system according to clause 1, wherein the processor performs additional operations, the additional operations comprising: preventing a malicious party who submits the submitted training data from accessing the machine learning model.

[0139] Clause 6. The system of clause 1, wherein the score is calculated as a Shapley value that quantifies the average marginal contribution of a participant relative to all subsets of the plurality of participants.

[0140] Clause 7. A system according to clause 1, wherein: the trusted computing environment is further configured to store multiple machine learning models, the multiple machine learning models including one machine learning model for each participant, and wherein the characteristic function is equal to the sum of the performance of all of the machine learning models plus the sum of the performance of each machine learning model for each individual participant.

[0141] Clause 8. A computer storage medium storing instructions that, when executed by a processor, perform operations comprising: executing a trusted computing environment configured to store a machine learning model and training data; storing criteria for accessing the machine learning model, the criteria indicating the performance of the machine learning model; calculating a score using the criteria and a characteristic function, the characteristic function being equal to the performance of the machine learning model plus the sum of the performance of the machine learning model for each individual participant; and controlling access to the machine learning model based on the calculated score.

[0142] Clause 9. A computer storage medium according to clause 8, wherein the criteria includes verification data, based on which only information relevant to the verification task is released through the machine learning model.

[0143] Clause 10. A computer storage medium according to clause 8, wherein the processor performs additional operations, the additional operations comprising: preventing another party who submits the training data to the trusted computing environment instead of submitting the standard from accessing the machine learning model.

[0144] Clause 11. A computer storage medium according to clause 8, wherein the processor performs additional operations, the additional operations comprising: using the calculated score to grant timed access to the machine learning model, the timing being related to the score.

[0145] Clause 12. A computer storage medium according to clause 8, wherein the processor performs additional operations, the additional operations comprising: preventing a malicious party who submits the training data that has already been submitted from accessing the machine learning model.

[0146] Clause 13. The computer storage medium of clause 8, wherein the score is calculated as a Shapley value that quantifies an average marginal contribution of a participant relative to all subsets of the plurality of participants.

[0147] Clause 14. A computer storage medium according to clause 8, wherein: the trusted computing environment is further configured to store a plurality of machine learning models, the plurality of machine learning models including a machine learning model for each participant, and wherein the characteristic function is equal to the sum of the performance of all of the machine learning models plus the sum of the performance of each machine learning model for each individual participant.

[0148] Clause 15. A method comprising: executing a trusted computing environment configured to store a machine learning model and training data; storing criteria for accessing the machine learning model on a memory, the criteria indicating performance of the machine learning model; calculating a score by a participating controller using the criteria and a characteristic function, the characteristic function being equal to the performance of the machine learning model plus the sum of the performance of the machine learning model for each individual participant; and controlling access to the machine learning model by an access controller based on the calculated score.

[0149] Clause 16. The method according to Clause 15 further includes: the access controller preventing another participant who submits the training data to the trusted computing environment instead of submitting the standard from accessing the machine learning model.

[0150] Clause 17. The method according to Clause 15 further includes: using the calculated score by the access controller to grant timed access to the machine learning model, wherein the timing is related to the calculated score.

[0151] Clause 18. The method of clause 15, wherein the method further comprises: the access controller preventing a malicious party who submits submitted training data from accessing the machine learning model.

[0152] Clause 19. The method of clause 15, wherein the score is calculated as a Shapley value that quantifies the average marginal contribution of a participant relative to all subsets of the plurality of participants.

[0153] Clause 20. The method according to Clause 15 further includes: storing multiple machine learning models by the trusted execution environment, wherein the multiple machine learning models include a machine learning model for each participant, and wherein the characteristic function is equal to the sum of the performance of all the machine learning models plus the sum of the performance of each machine learning model for each individual participant.

[0154] The disclosure presented herein also includes the subject matter described in the following clauses.

[0155] Clause 1. A system comprising: a processor; and a memory storing instructions that, when executed by the processor, perform operations comprising: receiving training data and criteria associated with a first party and a second party; calculating a score using the criteria and a characteristic function, the characteristic function being equal to the sum of the performance of a first machine learning model and the second machine learning model plus the sum of the performance gain of the first machine learning model for the first party and the second party and the performance gain of the second machine learning model for the first party and the second party, wherein the performance gain of the first machine learning model for the first party is equal to the performance of the first machine learning model based on the training data associated with the first party and the second party minus the performance of the first machine learning model based on the training data associated only with the first party, wherein the performance gain of the first machine learning model for the second party is equal to the performance of the first machine learning model based on the training data associated with the first party and the second party, The performance of the first machine learning model based on the training data associated with the first participant and the second participant is minus the performance of the first machine learning model based on the training data associated only with the second participant, wherein the performance gain of the second machine learning model for the first participant is equal to the performance of the second machine learning model based on the training data associated with the first participant and the second participant minus the performance of the second machine learning model based on the training data associated only with the first participant, wherein the performance gain of the second machine learning model for the second participant is equal to the performance of the second machine learning model based on the training data associated with the first participant and the second participant minus the performance of the second machine learning model based on the training data associated only with the second participant; and controlling access to the first machine learning model and the second machine learning model based on the calculated scores.

[0156] Clause 2. A system according to clause 1, wherein the criteria associated with the first party includes first verification data, and based on the first verification data, a portion of the training data associated with the first party is released through the first machine learning model.

[0157] Clause 3. A system according to clause 2, wherein the criteria associated with the first party includes second verification data, and based on the second verification data, another portion of the training data associated with the first party is released through the second machine learning model.

[0158] Clause 4. A system according to clause 3, wherein the criteria associated with the second party includes third verification data, based on which a portion of the training data associated with the second party is released through the first machine learning model.

[0159] Clause 5. A system according to clause 4, wherein the criteria associated with the second party includes fourth verification data, based on which another portion of the training data associated with the second party is released through the second machine learning model.

[0160] Clause 6. A system according to clause 3, wherein the portion of the training data associated with the first party remains private to the second machine learning model, and the other portion of the training data associated with the first party remains private to the first machine learning model.

[0161] Clause 7. A system according to clause 5, wherein the portion of the training data associated with the second party remains private to the second machine learning model, and the other portion of the training data associated with the second party remains private to the first machine learning model.

[0162] Clause 8. A method comprising: receiving training data and criteria associated with a first party and a second party; calculating a score using the criteria and a characteristic function, the characteristic function being equal to the sum of the performance of a first machine learning model and the second machine learning model plus the sum of the performance gain of each machine learning model for each individual party in the first party and the second party, wherein the performance gain for a machine learning model for a party is equal to the performance of the machine learning model based on the training data associated with the first party and the second party minus the performance of the machine learning model based on the training data associated only with the party; and controlling access to the first machine learning model and the second machine learning model based on the calculated score.

[0163] Clause 9. A method according to clause 8, wherein the criteria associated with the first party include first verification data, based on which a first portion of the training data associated with the first party is published through the first machine learning model.

[0164] Clause 10. A method according to clause 9, wherein the criteria associated with the first party includes second verification data, and based on the second verification data, a second portion of the training data associated with the first party is released through the second machine learning model.

[0165] Clause 11. A method according to clause 10, wherein the criteria associated with the second party include third verification data, based on which the first portion of the training data associated with the second party is published through the first machine learning model.

[0166] Clause 12. A method according to clause 11, wherein the criteria associated with the second party include fourth verification data, based on which a second portion of the training data associated with the second party is published through the second machine learning model.

[0167] Clause 13. A method according to clause 10, wherein the first portion of the training data associated with the first party remains private to the second machine learning model, and the second portion of the training data associated with the first party remains private to the first machine learning model.

[0168] Clause 14. A method according to clause 12, wherein the first portion of the training data associated with the second party remains private to the second machine learning model, and the second portion of the training data associated with the second party remains private to the first machine learning model.

[0169] Item 15. A computer storage medium storing instructions that, when executed by a processor, perform operations comprising: receiving training data and criteria associated with a first participant and a second participant; calculating a score using the criteria and a characteristic function, the characteristic function being equal to the sum of the performance of a first machine learning model and a second machine learning model plus the sum of the performance gain of the first machine learning model for the first participant and the second participant and the performance gain of the second machine learning model for the first participant and the second participant, wherein the performance gain of the machine learning model for a participant is equal to the performance of the machine learning model based on the training data associated with the first participant and the second participant minus the performance of the machine learning model based on the training data associated only with the participant; and controlling access to the first machine learning model and the second machine learning model based on the calculated score.

[0170] Clause 16. A computer storage medium according to clause 15, wherein the criteria associated with the first party includes first verification data, and based on the first verification data, a first portion of the training data associated with the first party is released through the first machine learning model.

[0171] Clause 17. A computer storage medium according to clause 16, wherein the criteria associated with the first party includes second verification data, and based on the second verification data, a second portion of the training data associated with the first party is released through the second machine learning model.

[0172] Clause 18. A computer storage medium according to clause 17, wherein the criteria associated with the second party includes third verification data, and based on the third verification data, the first portion of the training data associated with the second party is released through the first machine learning model.

[0173] Clause 19. A computer storage medium according to clause 18, wherein the criteria associated with the second party includes fourth verification data, and based on the fourth verification data, a second portion of the training data associated with the second party is released through the second machine learning model.

[0174] Clause 20. A computer storage medium according to clause 19, wherein the first portion of the training data associated with the first party remains private to the second machine learning model, and the second portion of the training data associated with the first party remains private to the first machine learning model, wherein the first portion of the training data associated with the second party remains private to the second machine learning model, and the second portion of the training data associated with the second party remains private to the first machine learning model.

[0175] The terms 'computer' or 'computing-based device' are used herein to refer to any device having processing capabilities, such that the device executes instructions. Those skilled in the art will recognize that such processing capabilities are incorporated into many different devices, and thus the terms 'computer' or 'computing-based device' each include personal computers (PCs), servers, mobile phones (including smartphones), tablet computers, set-top boxes, media players, game consoles, personal digital assistants, wearable computers, and many other devices.

[0176] In some examples, the methods described herein are performed by software in machine-readable form on a tangible storage medium (e.g., in the form of a computer program, including computer program code means adapted to perform all of the operations of one or more of the methods described herein when the program is run on a computer and where the computer program may be embodied on a computer-readable medium). The software is suitable for execution on a parallel processor or a serial processor such that the method operations may be performed in any suitable order or simultaneously.

[0177] This recognizes that software is a valuable, separately tradable commodity. It is intended to cover software that runs on or controls "virtual" or standard hardware to perform a desired function. It is also intended to cover software that "describes" or defines the configuration of hardware, such as HDL (Hardware Description Language) software used to design silicon chips or to configure general-purpose programmable chips to perform a desired function.

[0178] Those skilled in the art will recognize that the storage devices for storing program instructions are optionally distributed over a network. For example, a remote computer can store an example of a process described as software. A local or terminal computer can access the remote computer and download some or all of the software to run the program. Alternatively, the local computer can download software fragments as needed, or execute some software instructions at a local terminal and execute some software instructions at a remote computer (or computer network). Those skilled in the art will also recognize that, by utilizing conventional techniques known to those skilled in the art, all or part of the software instructions can be executed by a dedicated circuit such as a digital signal processor (DSP), a programmable logic array, etc.

[0179] It will be apparent to one skilled in the art that any range or device value given herein may be expanded or modified without losing the effect sought.

[0180] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

[0181] It should be understood that the benefits and advantages described above may relate to one embodiment or may relate to several embodiments. The embodiments are not limited to embodiments that solve any or all of the problems described or embodiments that have any or all of the benefits and advantages described. It should be further understood that references to 'an' item refer to one or more of those items.

[0182] The operations of the methods described herein may be performed in any suitable order or concurrently where appropriate. Additionally, individual blocks may be deleted from any of the methods without departing from the scope of the subject matter described herein. Aspects of any of the examples described above may be combined with aspects of any other example described to form further examples without losing the desired effect.

[0183] The term 'comprising' is used herein to mean including the identified method blocks or elements, but that such blocks or elements do not comprise an exclusive list and the method or apparatus may contain additional blocks or elements.

[0184] The term 'subset' is used herein to refer to a proper subset, such that a subset of a set does not include all elements of the set (ie, at least one of the elements of the set is missing from the subset).

[0185] It should be understood that the above description is given only as an example and that various modifications may be made by those skilled in the art. The above description, examples, and data provide a complete description of the structure and use of the exemplary embodiments. Although various embodiments have been described above with a certain degree of particularity or with reference to one or more individual embodiments, those skilled in the art may make various modifications to the disclosed embodiments without departing from the scope of this specification.

Claims

1. A system comprising: processor; as well as A memory storing instructions, wherein the instructions, when executed by the processor, perform operations, the operations comprising: executing a trusted computing environment configured to store machine learning models and training data; storing criteria for accessing the machine learning model, the criteria being indicative of performance of the machine learning model; calculating a score using the criteria and a characteristic function, the characteristic function being equal to the sum of the performance of the machine learning model plus the performance of the machine learning model for each individual participant; and Access to the machine learning model is controlled based on the calculated score.

2. The system of claim 1 , wherein the criteria include validation data, based on which only information relevant to the validation task is released by the machine learning model.

3. The system of claim 1 , wherein the processor performs additional operations comprising: Another party that submits the training data to the trusted computing environment instead of the standard is prevented from accessing the machine learning model.

4. The system of claim 1 , wherein the processor performs additional operations comprising: The calculated score is used to grant access to the machine learning model at a time, the timing being related to the score.

5. The system of claim 1 , wherein the processor performs additional operations comprising: Blocking access to the machine learning model by a malicious party that submitted the submitted training data.

6. A computer storage medium storing instructions, wherein when executed by a processor, the instructions perform operations comprising: executing a trusted computing environment configured to store machine learning models and training data; storing criteria for accessing the machine learning model, the criteria being indicative of performance of the machine learning model; calculating a score using the criteria and a characteristic function, the characteristic function being equal to the sum of the performance of the machine learning model plus the performance of the machine learning model for each individual participant; as well as Access to the machine learning model is controlled based on the calculated score.

7. A method comprising: executing a trusted computing environment configured to store machine learning models and training data; storing on a memory a criterion for accessing the machine learning model, the criterion being indicative of performance of the machine learning model; calculating, by a participating controller, a score using the criteria and a characteristic function, the characteristic function being equal to the sum of the performance of the machine learning model plus the performance of the machine learning model for each individual participant; as well as Access to the machine learning model is controlled by an access controller based on the calculated score.

8. A system comprising: processor; as well as A memory storing instructions, wherein the instructions, when executed by the processor, perform operations, the operations comprising: receiving training data and criteria associated with a first party and a second party; Calculate the score using the criteria and the feature function, The characteristic function is equal to the sum of the performance of the first machine learning model and the second machine learning model plus the sum of the performance gain of the first machine learning model for the first participant and the second participant and the performance gain of the second machine learning model for the first participant and the second participant, wherein the performance gain of the first machine learning model for the first party is equal to the performance of the first machine learning model based on the training data associated with the first party and the second party minus the performance of the first machine learning model based on the training data associated with only the first party, wherein the performance gain of the first machine learning model for the second party is equal to the performance of the first machine learning model based on the training data associated with the first party and the second party minus the performance of the first machine learning model based on the training data associated with only the second party, wherein the performance gain of the second machine learning model for the first party is equal to the performance of the second machine learning model based on the training data associated with the first party and the second party minus the performance of the second machine learning model based on the training data associated with only the first party, wherein the performance gain of the second machine learning model for the second party is equal to the performance of the second machine learning model based on the training data associated with the first party and the second party minus the performance of the second machine learning model based on the training data associated with only the second party; and Access to the first machine learning model and the second machine learning model is controlled based on the calculated scores.

9. A method comprising: receiving training data and criteria associated with a first party and a second party; Calculate the score using the criteria and the feature function, The characteristic function is equal to the sum of the performance of the first machine learning model and the second machine learning model plus the sum of the performance gain of each machine learning model for each individual participant of the first participant and the second participant, wherein the performance gain of the machine learning model for a party is equal to the performance of the machine learning model based on the training data associated with the first party and the second party minus the performance of the machine learning model based on the training data associated with only the party; and Access to the first machine learning model and the second machine learning model is controlled based on the calculated scores.

10. A computer storage medium storing instructions, wherein when executed by a processor, the instructions perform operations comprising: receiving training data and criteria associated with a first party and a second party; Calculate the score using the criteria and the feature function, The characteristic function is equal to the sum of the performance of the first machine learning model and the second machine learning model plus the sum of the performance gain of the first machine learning model for the first participant and the second participant and the performance gain of the second machine learning model for the first participant and the second participant, wherein the performance gain of the machine learning model for a party is equal to the performance of the machine learning model based on the training data associated with the first party and the second party minus the performance of the machine learning model based on the training data associated with only the party; and Access to the first machine learning model and the second machine learning model is controlled based on the calculated scores.

Images