Risk calculation device, risk calculation method, and risk calculation program

Abstract

This risk calculation device constructs a plurality of first shadow models using a predetermined dataset. Then, the risk calculation device selects, from a set of sample data, pieces of sample data for which the average loss of the plurality of first shadow models is sufficiently large and the variance thereof is small. Subsequently, the risk calculation device constructs a plurality of second shadow models using the selected pieces of sample data and the dataset. Next, the risk calculation device calculates, for the pieces of sample data, a distance between the distribution of losses of the plurality of first shadow models and the distribution of losses of the plurality of second shadow models. Then, the risk calculation device selects sample data for which the distance is equal to or larger than a predetermined threshold value as sample data (target sample) to be used for calculating a privacy risk of a machine learning model.

Description

[Technical Field]

[0001] This invention relates to a risk calculation device, a risk calculation method, and a risk calculation program for calculating the privacy risk of machine learning models. [Background technology]

[0002] Machine learning techniques, such as Deep Neural Networks (DNNs), have been criticized for posing privacy risks due to their ability to easily remember training data. Specifically, it has been shown that it is possible to estimate whether or not certain data was included in the training data from the output of a trained model. Therefore, when handling data that users do not want others to know, such as medical data or web browsing history, consideration for privacy risks is necessary.

[0003] Conventionally, methods have been proposed to calculate privacy risk based on the success rate of attacks that target trained models to determine whether certain data is included in the training data (see Non-Patent Documents 1, 2, and 3). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Jagielski, M. et al.: Auditing Differentially Private Machine Learning: How Private is Private SGD?, arXiv preprint arXiv:2006.07709 (2020). [Non-Patent Document 2] Nasr, M. et al.: Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning, arXiv preprint arXiv:2101.04535 (2021). [Non-Patent Document 3] Malek Esmaeili, M. et al.: Antipodes of Label Differential Privacy: PATE and ALIBI, Proceedings of the 35th Advances in Neural Information Processing Systems, pp. 6934-6945 (2021). [Overview of the project] [Problems that the invention aims to solve]

[0005] However, conventional methods for calculating privacy risk have only proposed methods based on weak or unrealistic attacks, which has the problem of underestimating privacy risk in realistic settings. Calculating privacy risk based on weak attacks involves, for example, calculating the average privacy risk using randomly selected target samples. Unrealistic attacks, on the other hand, include attacks that assume the attacker has access to a model under training or attacks that assume the attacker can manipulate the training data.

[0006] Therefore, the present invention aims to solve the aforementioned problems and to accurately calculate privacy risks in realistic settings for machine learning models. [Means for solving the problem]

[0007] To solve the above problems, the present invention includes a first model construction unit that constructs a plurality of first shadow models using a predetermined data set, a first selection unit that selects, from sample data, the sample data for which the average of the losses of the plurality of first shadow models is greater than or equal to a predetermined value and the variance is less than or equal to a predetermined value, a second model construction unit that constructs a plurality of second shadow models using the selected sample data and the data set, a distance calculation unit that calculates the distance between the loss distribution of the plurality of first shadow models and the loss distribution of the plurality of second shadow models for the sample data, a second selection unit that selects, as sample data to be used for calculating the privacy risk of a machine learning model, the sample data for which the distance is greater than or equal to a predetermined threshold value, and an output processing unit that outputs the selected sample data.

Effect of the Invention

[0008] According to the present invention, the privacy risk in a realistic setting of a machine learning model can be calculated highly (correctly).

Brief Description of the Drawings

[0009] [Figure 1] FIG. 1 is a diagram for explaining the outline of a risk calculation device. [Figure 2] FIG. 2 is a diagram showing a configuration example of a risk calculation device. [Figure 3] FIG. 3 is a flowchart showing an example of a processing procedure executed by a risk calculation device. [Figure 4] FIG. 4 is a flowchart showing an example of calculating privacy risk to which a risk calculation device is applied. [Figure 5] FIG. 5 is a diagram showing a computer that executes a risk calculation program.

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments for implementing the present invention will be described with reference to the drawings. The present invention is not limited to these embodiments.

[0011] [Overview] First, the overview of the risk calculation device of this embodiment will be described using FIG. 1. The risk calculation device selects sample data (target sample) used for calculating the privacy risk of a learned model (machine learning model) as follows.

[0012] First, the risk calculation device constructs a plurality of shadow models (first shadow models) using a pre-prepared dataset. Next, the risk calculation device inputs sample data (data different from the dataset) into each of the constructed plurality of first shadow models. Then, the risk calculation device calculates the loss of each of the constructed plurality of first shadow models, and selects sample data for which the average of the calculated losses is sufficiently large and the variance is small (selection 1).

[0013] Next, the risk calculation device constructs a plurality of shadow models (second shadow models) using the data obtained by adding the sample data selected in selection 1 to the above dataset. Then, the risk calculation device calculates the loss of each of the plurality of second shadow models with respect to the sample data.

[0014] Thereafter, the risk calculation device selects, as the target sample, sample data for which the difference between the loss distribution of each of the plurality of first shadow models and the loss distribution of each of the plurality of second shadow models becomes large (selection 2).

[0015] In other words, sample data where the average loss of multiple shadow models is sufficiently large and the variance is small can be considered sample data with a high privacy risk. Furthermore, if adding sample data to the training data of a shadow model significantly alters the distribution of the model's loss, that sample data can be considered data that is likely to be predictable. Therefore, that sample data can be considered data with a high privacy risk.

[0016] Therefore, the risk calculation device first selects sample data with a sufficiently large average loss and small variance among multiple shadow models as candidate target samples (Selection 1). Next, the risk calculation device selects from the candidate target samples those that, when added to the training data of a shadow model, significantly change the distribution of the loss of that shadow model (Selection 2).

[0017] In this way, the risk calculation device can select sample data with a high privacy risk as the target sample. Then, by using the above target sample to calculate the privacy risk of the machine learning model, the risk calculation device can accurately calculate the privacy risk.

[0018] [Example Configuration] Next, an example of the configuration of the risk calculation device 10 will be described using Figure 2. The risk calculation device 10 includes, for example, an input / output unit 11, a storage unit 12, and a control unit 13.

[0019] The input / output unit 11 is an interface that handles the input and output of various types of data. For example, the input / output unit 11 accepts input such as a dataset and sample data. The dataset is used to construct the first shadow model and the second shadow model. The sample data is candidate data (target samples) used to calculate the privacy risk of the machine learning model.

[0020] The memory unit 12 stores data, programs, etc., that are referenced when the control unit 13 performs various processes. The memory unit 12 is implemented by semiconductor memory elements such as RAM (Random Access Memory) and flash memory, or by storage devices such as hard disks and optical discs.

[0021] For example, the memory unit 12 stores datasets, sample data, etc., that are input via the input / output unit 11. Also, for example, the memory unit 12 stores parameters of the first shadow model and parameters of the second shadow model constructed by the control unit 13.

[0022] The control unit 13 is responsible for controlling the entire risk calculation device 10. The functions of the control unit 13 are realized, for example, by the CPU (Central Processing Unit) executing a program stored in the memory unit 12.

[0023] The control unit 13 includes, for example, a first model construction unit 131, a first selection unit 132, a second model construction unit 133, a distance calculation unit 134, a second selection unit 135, a risk calculation unit 136, and an output processing unit 137. The label operation unit 138 and the noise addition unit 139, shown by dashed lines, may or may not be included; the cases in which they are included will be described later.

[0024] The first model building unit 131 uses the above dataset as training data to build multiple first shadow models. The settings used by the first model building unit 131 when building the first shadow models are the same as the settings for the machine learning model used to calculate privacy risk.

[0025] The first selection unit 132 inputs sample data into multiple first shadow models constructed by the first model building unit 131, and selects sample data in which the average loss of the multiple first shadow models is sufficiently large and the variance is small.

[0026] For example, first, the first selection unit 132 calculates the loss of each of the first shadow models for the sample data. Next, the first selection unit 132 calculates the average and variance of the calculated losses. Then, the first selection unit 132 selects the sample data for which s shown in the following equation (1) outi is small.

[0027]

Number

[0028] Note that in Equation (1), μ outi is the average of the losses of a plurality of first shadow models for the sample data (x i , y i ), and σ 2 outi is the variance of the losses of a plurality of first shadow models for the sample data (x i , y i ). k is a constant for a sufficiently large loss. For example, k is about 20. The value of this k may be, for example, a value determined based on the analysis result of samples vulnerable to attacks.

[0029] Alternatively, the first selection unit 132 may select sample data for which the average of the losses of a plurality of first shadow models is large and the variance is small without using the above constant k. In that case, the first selection unit 132 selects the sample data for which s shown in the following equation (2) outi is small.

[0030]

Number

[0031] For example, the first selection unit 132 sequentially selects a predetermined number (for example, about 100 - 1000) of sample data from the sample data for which s outi calculated by the above equation (1) or equation (2) is small. Also, the first selection unit 132 may select sample data for which the above s outi is less than or equal to a predetermined value.

[0032] Furthermore, the first selection unit 132 may, for example, select sample data in which the average of the losses of multiple first shadow models for the sample data is greater than or equal to a predetermined value, and the variance of the losses is less than or equal to a predetermined value.

[0033] The second model building unit 133 constructs multiple second shadow models using the sample data selected by the first selection unit 132 and the dataset (the dataset used to construct the first shadow model) as training data. The settings used by the second model building unit 133 when constructing the second shadow models are the same as the settings for the machine learning model to which the privacy risk is calculated.

[0034] The distance calculation unit 134 calculates how much the loss distribution of multiple first shadow models and the loss distribution of multiple second shadow models differ for the sample data.

[0035] For example, the distance calculation unit 134 calculates the distance d between the distribution of losses of multiple first shadow models and the distribution of losses of multiple second shadow models for the sample data selected by the first selection unit 132, based on the following equation (3). i Calculate.

[0036]

number

[0037] Note that in equation (3), μ ini This is sample data (x i ,y i ) is the average of the losses of multiple second shadow models, σ 2 ini This is sample data (x i ,y i This is the variance of the losses of multiple second shadow models against ).

[0038] The second selection unit 135 calculates the distance d between the loss distribution of a plurality of first shadow models calculated by the distance calculation unit 134 and the loss distribution of a plurality of second shadow models. i Select the largest sample data.

[0039] For example, the second selection unit 135 is the distance d mentioned above. i A predetermined number of sample data (for example, 1 to 100) are selected in order from the largest sample data. The second selection unit 135 also selects the above distance d i You may select sample data where the value is greater than or equal to a predetermined value.

[0040] The risk calculation unit 136 calculates the privacy risk of the machine learning model using the sample data (target sample) selected by the second selection unit 135.

[0041] For example, data containing the target sample (D1) and data not containing the target sample (D0) are prepared. The risk calculation unit 136 then plays a game 1000 times to guess whether or not the data containing the target sample (D1) was used to train the machine learning model, based on the following equation (4). The risk calculation unit 136 then calculates the privacy risk of the machine learning model from the accuracy rate of the game.

[0042]

number

[0043] The output processing unit 137 outputs the processing results from the control unit 13. For example, the output processing unit 137 outputs the results of the risk calculation unit 136's calculation of the privacy risk of the machine learning model.

[0044] Such a risk calculation device 10 can accurately calculate the privacy risk of a machine learning model.

[0045] Note that the calculation of privacy risk using the target sample selected by the second selection unit 135 may be performed by a device other than the risk calculation device 10 (an external device). In this case, the risk calculation device 10 outputs the target sample selected by the second selection unit 135 to the external device. The external device then calculates the privacy risk using the target sample.

[0046] [Example of processing procedure] Next, an example of the processing procedure performed by the risk calculation device 10 will be explained using Figure 3. First, the first model construction unit 131 of the risk calculation device 10 constructs multiple first shadow models using the dataset (S1).

[0047] After S1, the first selection unit 132 selects sample data (S2) based on the mean and variance of the losses of the multiple first shadow models constructed in S1. For example, the first selection unit 132 selects sample data from the sample data in which the mean of the losses of the multiple first shadow models is sufficiently large and the variance is small.

[0048] After S2, the second model building unit 133 builds multiple second shadow models using the sample data and dataset (the dataset used to build the first shadow model) selected in S2 (S3).

[0049] After S3, the distance calculation unit 134 calculates the distance between the loss distribution of the multiple first shadow models constructed in S1 and the loss distribution of the multiple second shadow models constructed in S3 (S4).

[0050] After S4, the second selection unit 135 selects sample data (target samples) whose distance calculated in S4 is greater than or equal to a predetermined threshold (S5). Then, the risk calculation unit 136 calculates the privacy risk of the machine learning model using the sample data (target samples) selected in S5 (S6). Finally, the output processing unit 137 outputs the privacy risk calculation result obtained in S6 (S7).

[0051] By performing the above processing, the risk calculation device 10 can accurately calculate the privacy risk under realistic settings of the machine learning model.

[0052] [Other embodiments] The risk calculation device 10 may also include a label manipulation unit 138 (see Figure 2). The label manipulation unit 138 creates new sample data by performing label operations on the sample data.

[0053] For example, the label manipulation unit 138 extracts new sample data (x,y) from the sample data (x,y) (where y is the label for x). p Create ). However, y p ≠y. Here, the label manipulation unit 138 generates new sample data (x,y p When creating the ), sample data for all classes other than the correct answer (y) may be created. For example, if the shadow model is a model that classifies the input data into 10 classes, the label manipulation unit 138 will create y p Alternatively, you could create sample data with nine labels other than the correct answer (y).

[0054] Then, the first selection unit 132 selects sample data from the sample data to which new sample data has been added by the label manipulation unit 138, in which the average loss of multiple first shadow models is sufficiently large and the variance is small. Subsequently, the second selection unit 135 selects a target sample from the sample data selected by the first selection unit 132.

[0055] Furthermore, the risk calculation device 10 may also include a noise addition unit 139 (see Figure 2). The noise addition unit 139 creates sample data that is vulnerable to attack by adding noise to the sample data.

[0056] For example, the noise addition unit 139 adds noise to the sample data such that the average loss of multiple first shadow models is sufficiently large and the variance is small. For example, the noise addition unit 139 optimizes the noise so that the losses shown in equations (5) and (6) below are small. Then, the noise addition unit 139 adds the optimized noise to the sample data.

[0057]

number

number

[0058] Furthermore, the noise addition unit 139 may add conditions such that the added noises are not similar to each other. For example, the noise addition unit 139 has a loss L as shown in the following equation (7). sim Noise is added to reduce x. Note that in equation (7), x i t This is sample data with noise added at step t of the noise addition optimization.

[0059]

number

[0060] In this way, the noise addition unit 139 creates sample data that is vulnerable to attacks, allowing the risk calculation device 10 to select target samples with a higher privacy risk. As a result, the risk calculation device 10 can calculate the privacy risk of machine learning models in realistic settings more accurately.

[0061] [Examples of application] Next, an example of the application of the risk calculation device 10 will be explained using Figure 4. For example, the administrator of the risk calculation device 10 designs a neural network (NN) to be used for calculating privacy risks (S11). For example, the optimization method DP-SGD (Differentially Private Stochastic Gradient Descent) is used to train an NN that satisfies Differential Privacy.

[0062] Next, the risk calculation device 10 selects a target sample from the sample data (S12). Then, the risk calculation device 10 calculates the privacy risk using the target sample selected in S12 (S13). For example, the risk calculation device 10 uses the target sample selected in S12 to play a game for calculating the privacy risk of conventional technology 1000 times, and calculates the privacy risk of the machine learning model from the accuracy rate.

[0063] [System configuration, etc.] Furthermore, the components of each part shown in the diagram are functional concepts and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown in the diagram, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions. Moreover, all or any part of the processing functions performed by each device can be realized by a CPU and the program executed on that CPU, or by hardware using wired logic.

[0064] Furthermore, among the processes described in the embodiments described above, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified.

[0065] [program] The aforementioned risk calculation device 10 can be implemented by installing a program (risk calculation program) as packaged software or online software on a desired computer. For example, by having the above program run on an information processing device, the information processing device can function as the risk calculation device 10. The information processing device referred to here includes mobile communication terminals such as smartphones, mobile phones and PHS (Personal Handyphone System), as well as terminals such as PDA (Personal Digital Assistant).

[0066] Figure 5 shows an example of a computer running a risk calculation program. Computer 1000 has, for example, memory 1010 and a CPU 1020. Computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0067] Memory 1010 includes ROM (Read Only Memory) 1011 and RAM (Random Access Memory) 1012. ROM 1011 stores, for example, a boot program such as BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. For example, a removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.

[0068] The hard disk drive 1090 stores, for example, the OS 1091, application program 1092, program module 1093, and program data 1094. That is, the program that defines each process executed by the risk calculation device 10 is implemented as a program module 1093 in which executable code for a computer is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing a process similar to the functional configuration of the risk calculation device 10 is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0069] Furthermore, the data used in the processing of the above-described embodiment is stored as program data 1094 in, for example, memory 1010 or hard disk drive 1090. The CPU 1020 then reads the program module 1093 and program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as needed and executes them.

[0070] Furthermore, the program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090; for example, they may be stored in a removable storage medium and read by the CPU 1020 via a disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via a network interface 1070. [Explanation of symbols]

[0071] 10. Risk calculation device 11 Input / output section 12 Storage section 13 Control Unit 131 First Model Construction Section 132 First Selection Section 133 Second Model Building Section 134 Distance Calculation Unit 135 Second Selection Section 136 Risk Calculation Section 137 Output Processing Unit 138 Label operation unit 139 Noise Addition Section

Claims

1. A first model building unit that constructs multiple first shadow models using a predetermined dataset, A first selection unit selects sample data from the sample data such that the average loss of multiple first shadow models is greater than or equal to a predetermined value and the variance is less than or equal to a predetermined value. A second model construction unit that constructs multiple second shadow models using the selected sample data and the dataset, A distance calculation unit that calculates the distance between the loss distributions of a plurality of first shadow models and the loss distributions of a plurality of second shadow models for the sample data, A second selection unit selects the sample data for which the distance is greater than or equal to a predetermined threshold as sample data to be used for calculating the privacy risk of the machine learning model. An output processing unit that outputs the selected sample data and A risk calculation device characterized by being equipped with the following features.

2. Risk calculation unit calculates the privacy risk of the machine learning model using the output sample data. The risk calculation device according to claim 1, further comprising the features described above.

3. A noise addition unit adds noise to the sample data selected by the first selection unit such that the average loss of the plurality of first shadow models increases and the variance decreases. The risk calculation device according to claim 1, further comprising the features described above.

4. The system further includes a label manipulation unit that adds new sample data by performing label operations on the aforementioned sample data. The first selection unit described above is, From the sample data to which the newly added sample data is added, select the sample data in which the average loss of multiple first shadow models is greater than or equal to a predetermined value, and the variance is less than or equal to a predetermined value. The risk calculation device according to feature 1.

5. A risk calculation method performed by a risk calculation device, The process involves constructing multiple first shadow models using a predetermined dataset, A step of selecting sample data from which the average loss of multiple first shadow models is greater than or equal to a predetermined value and the variance is less than or equal to a predetermined value, The process involves constructing multiple second shadow models using the selected sample data and the dataset, A step of calculating the distance between the loss distribution of a plurality of first shadow models and the loss distribution of a plurality of second shadow models with respect to the sample data, A step of selecting the sample data for which the distance is greater than or equal to a predetermined threshold as sample data to be used for calculating the privacy risk of the machine learning model, A step of outputting the selected sample data and A risk calculation method characterized by including the following.

6. The process involves constructing multiple first shadow models using a predetermined dataset, A step of selecting sample data from which the average loss of multiple first shadow models is greater than or equal to a predetermined value and the variance is less than or equal to a predetermined value, The process involves constructing multiple second shadow models using the selected sample data and the dataset, A step of calculating the distance between the loss distribution of a plurality of first shadow models and the loss distribution of a plurality of second shadow models with respect to the sample data, A step of selecting the sample data for which the distance is greater than or equal to a predetermined threshold as sample data to be used for calculating the privacy risk of the machine learning model, A step of outputting the selected sample data and A risk calculation program that is executed by a computer.

Images