Evaluation device, evaluation method, and program

The evaluation device in federated learning assesses the importance of leaked information by extracting key data, calculating parameter changes, and simulating attacks, providing a clear evaluation of data leakage risk, thus addressing the uncertainty in information leakage and aiding decision-making in participating in federated learning.

WO2024247166A1PCT designated stage expired Publication Date: 2024-12-05NT T INC
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2023/020293
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-31
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

In federated learning, participants are unaware of the amount of information that will be leaked, and it is unclear whether leaked information is important, making it difficult to assess the impact of data leakage during model sharing.

Method used

An evaluation device and method that extracts important information from learning data, generates a list, calculates the amount of change in model parameters, performs a restoration attack using dummy data to estimate the learning data, and evaluates the degree of leakage, providing an evaluation value to assess the importance of leaked information.

Benefits of technology

Enables the evaluation of the extent of important information leakage in federated learning, allowing participants to assess their data leakage risk and the effectiveness of defensive measures, thereby informing participation decisions and improving security.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2023020293_05122024_PF_FP_ABST
    Figure JP2023020293_05122024_PF_FP_ABST
Patent Text Reader

Abstract

The present invention provides an evaluation device, etc., for evaluating how much important information leaks. This evaluation device includes: a list generation unit that extracts important information from training data, lists the important information, and generates an important information list; a model training unit that trains a model on the basis of the training data, updates model parameters, and obtains an amount of change; a restoration attack unit that performs a restoration attack using the amount of change and obtains an estimated value of the training data; and an evaluation unit that evaluates a degree of data leakage due to the restoration attack on the basis of the important information list and the estimated value of the training data.
Need to check novelty before this filing date? Find Prior Art

Description

Evaluation device, evaluation method, and program

[0001] The present invention relates to a technique for evaluating the degree of data leakage against a reconstruction attack in federated learning.

[0002] By utilizing information such as emails and purchase histories stored on personal devices, corporate research materials and IoT information, and hospital diagnostic information, many benefits can be achieved, such as matching, automatic control, and AI medical treatment. However, due to concerns about information security and leaks, this information cannot be utilized. Federated learning, which allows learning to be carried out in a distributed manner, is well known as a solution to this problem.

[0003] Federated learning allows for learning a model that has learned the characteristics of all data while ensuring data security in an environment where data is stored in a distributed manner (see Non-Patent Documents 1 and 2).

[0004] H. Brendan McMahan and Eider Moore and Daniel Ramage and Seth Hampson and Blaise Aguera y Arcas, "Communication-Efficient Learning of Deep Networks from Decentralized Data", [online], arXiv:1602.05629v4 [cs.LG] 26 Jan 2023, [Retrieved 2023 / 05 / 24], Internet<URL:https: / / arxiv.org / pdf / 1602.05629.pdf> Qinbin Li, Zeyi Wen, Zhaomin Wu, Sixu Hu, Naibo Wang, Yuan Li, Xu Liu, Bingsheng He, "A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection", [online], arXiv:1907.09693v7 [cs.LG] 5 Dec 2021, [Searched 2023 / 05 / 24], Internet<URL:https: / / arxiv.org / pdf / 1907.09693.pdf>

[0005] However, in federated learning, it is known that information can be leaked by attacking the shared model (References 1, 2).

[0006] (Reference 1) Ligeng Zhu, Zhijian Liu, Song Han, "Deep Leakage from Gradients", [online], arXiv:1906.08935v2 [cs.LG] 19 Dec 2019, [Retrieved 2023 / 05 / 24], Internet<https: / / arxiv.org / pdf / 1906.08935.pdf> (Reference 2) Mislav Balunovi, Dimitar I. Dimitrov, Nikola Jovanovic, Martin Vechev, "LAMP: Extracting Text from Gradients with Language Model Priors", [online], arXiv:2202.08827v2 [cs.LG] 19 Oct 2022, [Searched on 2023 / 05 / 24], Internet<https: / / arxiv.org / pdf / 2202.08827.pdf> Here, there is a problem that participants in federated learning do not know how much of their information will be leaked, and when information is leaked, they do not know whether the information is important or not.

[0007] An object of the present invention is to provide an evaluation device, an evaluation method, and a program for evaluating the extent to which important information is leaked.

[0008] In order to solve the above problems, according to one aspect of the present invention, an evaluation device includes a list generation unit that extracts important information from training data, lists it, and generates an important information list; a model training unit that trains a model based on the training data, updates model parameters, and determines a change amount; a restoration attack unit that uses the change amount to perform a restoration attack and determine an estimated value of the training data; and an evaluation unit that evaluates the degree of data leakage against the restoration attack based on the important information list and the estimated value of the training data.

[0009] The present invention provides an advantage in that it is possible to evaluate the degree to which important information is leaked in associative learning.

[0010] 1 is a functional block diagram of an evaluation device according to a first embodiment; a diagram showing an example of a processing flow of the evaluation device according to the first embodiment; a diagram showing an example of data stored in a storage unit; a diagram showing an example of an estimated value; a diagram showing an example of the number of occurrences of important information; a diagram for explaining w, t, and s; a diagram showing examples of training data and important information; a diagram showing an example of the configuration of a computer to which the present method is applied.

[0011] Hereinafter, an embodiment of the present invention will be described. In the drawings used in the following description, components having the same functions and steps performing the same processes are denoted by the same reference numerals, and redundant description will be omitted.

[0012] Before explaining the key points of the first embodiment, first, federated learning and recovery attacks will be explained.

[0013] (Federated Learning) Federated learning is a method of performing machine learning in a distributed manner without aggregating data.

[0014] In federated learning, the learning system includes a server device and N client devices, and performs machine learning in the following procedure.

[0015] (1) The server device distributes the model or model parameters to each client device.

[0016] (2) Each client device learns based on its own data (learning data), updates the model parameters, calculates the change ∇W (for example, gradient) of the model parameters, and transmits it to the server device.

[0017] (3) The server device receives the variation ∇W from the N client devices, integrates the received information, and updates the model parameters. For example, it takes a weighted average of the variation ∇W based on the amount of training data and updates the model parameters.

[0018] The model is trained by repeating steps (1) to (3) above.

[0019] By performing such processing, the amount of change ∇W of each client device is output to the outside, but the learning data of each client device is not output to the outside.

[0020] (Restoration Attack) In a restoration attack, it is possible to estimate, from the amount of change ∇W, the learning data of the client device that calculated the amount of change ∇W.

[0021] As explained in (2) above, the client device used by a normal participant learns based on its own learning data, updates the model parameters, and calculates the amount of change ∇W.

[0022] On the other hand, a client device used by a malicious attacker learns based on dummy data, updates the model parameters, and calculates the change ∇W' (e.g., gradient) of the model parameters. If the change ∇W is available, the client device used by the malicious attacker can estimate the learning data of the client devices used by normal participants by updating the dummy data so that the distance between the change ∇W and the change ∇W' becomes smaller (see, for example, Reference 1). <Key Points of the First Embodiment> The evaluation device of this embodiment is implemented on a client device included in a learning system that performs federated learning.

[0023] The evaluation device, like a normal client device, learns based on its own learning data, updates the model parameters, and calculates the amount of change ∇W.

[0024] Furthermore, the evaluation device extracts and lists important information contained in its own learning data, and generates an important information list.

[0025] Furthermore, the evaluation device learns based on dummy data, updates the model parameters, and obtains the variation ∇W'. The evaluation device obtains an estimate of its own learning data by updating the dummy data so that the distance between the variation ∇W and the variation ∇W' becomes smaller.

[0026] The evaluation device evaluates the degree of data leakage against a recovery attack based on the important information list and the estimated value, and outputs an evaluation value. For example, the evaluation device evaluates the degree of data leakage as being higher the more important information in the important information list is included in the estimated value.

[0027] The specific configuration of the evaluation device will be described below.

[0028] First Embodiment FIG. 1 is a functional block diagram of an evaluation device 100 according to a first embodiment, and FIG. 2 shows the processing flow thereof.

[0029] The evaluation device 100 includes an input unit 101 , a storage unit 110 , a list generation unit 120 , a model learning unit 130 , a restoration attack unit 140 , and an evaluation unit 150 .

[0030] The evaluation device 100 uses the training data to evaluate the degree of data leakage against a recovery attack and outputs an evaluation value.

[0031] For example, a client device on which the evaluation device is implemented may be a smartphone, a personal computer, or the like. When an application is downloaded from a server device to a smartphone and installed, the model and model parameters are distributed. The smartphone accumulates its own data obtained by using the application, and the accumulated data is used as training data.

[0032] The evaluation device 100 is a special device configured by loading a special program into a publicly known or dedicated computer having, for example, a central processing unit (CPU) and a main memory (RAM). The evaluation device 100 executes each process under the control of the central processing unit. Data input to the evaluation device 100 and data obtained in each process are stored in, for example, the main memory. The data stored in the main memory is read by the central processing unit as needed and used for other processes. At least a portion of each processing unit of the evaluation device 100 may be configured with hardware such as an integrated circuit. Each storage unit included in the evaluation device 100 may be configured with, for example, a main storage unit such as a RAM (Random Access Memory) or middleware such as a relational database or a key-value store. However, each storage unit does not necessarily have to be included within the evaluation device 100 itself; it may be configured with an auxiliary storage unit including a hard disk, optical disk, or semiconductor memory element such as a flash memory, and be configured external to the evaluation device 100.

[0033] Each part will be explained below.

[0034] <Input Unit 101 and Storage Unit 110> The storage unit 110 stores training data input via the input unit 101. FIG. 3 shows an example of data stored in the storage unit 110. The training data may be acquired externally or may be acquired based on processing within a client device in which the evaluation device 100 is implemented. The input unit 101 may be an input interface for acquiring information from the outside, a touch panel, a keyboard, a mouse, a camera, a microphone, or the like. For example, if the input unit 101 includes a touch panel, a keyboard, a mouse, or the like, text data (e.g., FIG. 3 ) input by operating these may be used as the training data, as well as data in various other formats. If the input unit 101 includes a camera, image data captured by the camera may be used as the training data. If the input unit 101 includes a microphone, audio data picked up by the microphone may be used as the training data. Furthermore, a combination of text data, image data, and audio data may be used as the training data.

[0035] <List generation unit 120> The list generation unit 120 retrieves the learning data from the storage unit 110, extracts important information from the learning data, lists it, generates an important information list (S120), and stores it in the storage unit 110 in association with the learning data (see Figure 3).

[0036] <Model Learning Unit 130> The model learning unit 130 retrieves learning data from the storage unit 110, and, like a normal client device, learns a model based on the learning data (S130), updates the model parameters, and calculates and outputs the amount of change ∇W.

[0037] <Restoration attack unit 140> The restoration attack unit 140 receives the change ∇W, performs M restoration attacks using the change ∇W (S140), and obtains and outputs an estimated value D=[d'(1),d'(2),...,d'(M)] of the learning data (S140). M is any integer equal to or greater than 1, and m=1, 2,...,M. For example, a restoration attack is performed by the following process.

[0038] The recovery attack unit 140 generates M pieces of dummy data d(m). As the dummy data d(m), for example, random numbers based on a standard normal distribution are used.

[0039] The restoration attack unit 140 learns the model based on dummy data d(m) instead of the training data, updates the model parameters, and finds the amount of change ∇W'.

[0040] Furthermore, the restoration attack unit 140 updates the dummy data d(m) so that the distance between the amount of change ∇W and the amount of change ∇W'(m) becomes smaller.

[0041] The restoration attack unit 140 repeats the calculation of the change amount ∇W' and the update of the dummy data until the change amount ∇W' and the update of the dummy data converge, and obtains the dummy data d'(m) after convergence as the estimated value d'(m) of the training data used when obtaining the normal model.

[0042] The recovery attack unit 140 performs processing for m=1, 2, ..., M, and outputs D=[d'(1), d'(2), ..., d'(M)].

[0043] <Evaluation unit 150> The evaluation unit 150 retrieves the important information list corresponding to the training data from the storage unit 110, and receives the output of the recovery attack unit 140 (the estimated value D of the training data = [d'(1), d'(2), ..., d'(M)]).

[0044] The evaluation unit 150 evaluates the degree of data leakage against a restoration attack based on the important information list and D = [d'(1), d'(2), ..., d'(M)] (S150) and outputs an evaluation value Score. For example, if the information included in D = [d'(1), d'(2), ..., d'(M)] matches or is similar to important information included in the important information list, it can be determined that data has been leaked by a restoration attack, and the degree of data leakage against a restoration attack is evaluated as high. Furthermore, the more the information included in D = [d'(1), d'(2), ..., d'(M)] matches or is similar to important information included in the important information list, the more data can be determined to have been leaked, and the higher the degree of data leakage against a restoration attack is evaluated.

[0045] An example in which the evaluation device 100 of the first embodiment is used in a natural language task will be described below.

[0046] (Example of Use in Natural Language Tasks) In the case of natural language tasks, the training data is words, character strings, sentences, etc. In the example of Fig. 3, the training data is sentences.

[0047] The list generation unit 120 extracts important words contained in the sentences that are the training data as important information, lists them, generates an important information list (for convenience, also referred to as a main list), and stores the list in the storage unit 110 in association with the training data (see FIG. 3). Syntactic analysis (see References 3 and 4) can be used to extract important words from sentences.

[0048] (Reference 3) "Python Tech - Python Learning Site [Python] Basic Usage of Syntax Analysis with CaboCha", [online], Python Tech, [Searched 2023 / 05 / 26], Internet<URL:https: / / tech.nkhn37.net / python-cabocha-syntactic-analysis / > (Reference 4) "Syntax Analysis: A Natural Language Processing Master's Course," [online], [searched 2023 / 05 / 26], Internet<URL:https: / / www.jnlp.org / nlp / %E6%A7%8B%E6%96%87%E8%A7%A3%E6%9E%90 / top> The list generation unit 120 may also be configured to extract important words contained in sentences that are training data as important information, divide them into meanings, list them, generate one or more important information lists (also referred to as sublists for convenience), and store them in the storage unit 110 in association with the training data (see FIG. 3). When dividing important words into meanings, semantic analysis, context analysis, etc. can be used.

[0049] The restoration attack unit 140 obtains an estimated value d'(m) of the training data restored from the dummy data by the restoration attack. Figure 4 shows an example of the estimated value d'(m) when the training data is "Alice and Ken have cancer."

[0050] The evaluation unit 150 counts the number of occurrences of important information included in the M estimated values ​​d'(m). FIG. 5 shows an example of the number of occurrences of important information. Furthermore, the evaluation unit 150 calculates an evaluation value Score from the number of occurrences of important information. For example, the following four calculation methods are possible:

[0051] (Calculation Method 1) Calculation Method 1 is a score calculation method that emphasizes that all important information can be restored. Specifically, the evaluation value Score is calculated using the following formula. Here, w is the number of restored words, t indicates which word in the most frequent occurrence count covers all the information in the important information list (main list), and s is the length of the original sentence (number of words). Figure 6 is a diagram to explain w, t, and s.

[0052] (Calculation Method 2) Calculation Method 2 is a score calculation method that places importance on the ability to restore important information. Specifically, the evaluation value Score is calculated using the following formula. where w i is each restored important information among the important information included in the important information list (main list), n is the number of important information included in the important information list (main list), and Count(w i ) is important information i is the number of occurrences of, and M is the number of recovery attacks.

[0053] (Calculation Method 3) Calculation Method 3 is a score calculation method that takes into account the meaning of the sentence and places importance on being able to restore important information that makes sense of part of the sentence. Specifically, the evaluation value Score is calculated using the following formula. Score=max (SubScore j ) where j indicates the sublist number and w j,i is each restored important information contained in the important information list (sublist) j, and n j is the number of important information items included in important information list (sublist) j.

[0054] (Calculation Method 4) Calculation Method 4 is a score calculation method that takes into account the meaning of the sentence and places importance on being able to restore important information that makes sense of each sentence. Specifically, the evaluation value Score is calculated using the following formula. where S is the number of sublists.

[0055] It should be noted that the important information list (sub-list) is not necessary when using only calculation method 1 or calculation method 2. The important information list (main list) is not necessary when using only calculation method 3 or calculation method 4.

[0056] Alternatively, two or more of calculation methods 1 to 4 may be employed to obtain two or more evaluation values ​​Score, and the two or more evaluation values ​​Score obtained may be weighted and added to obtain the final evaluation value Score. This configuration allows for flexible evaluation depending on the information that is emphasized.

[0057] Furthermore, in this example, if important information (word) in the important information list matches a word included in the estimated value d'(m), the number of occurrences is increased by one. However, it is also possible to calculate the similarity between important information (word) in the important information list and the estimated value d'(m), and if the similarity is greater than a predetermined threshold, it is determined that important information has appeared in the estimated value d'(m), and the number of occurrences is increased by one.

[0058] In this example, important information is measured in units of words, but it may also be measured in units of character strings or sentences. In this case, it is better to use similarity, since there are fewer cases where important information (character strings or sentences) in the important information list matches the character strings or sentences included in the estimated value d'(m). (Example of use in image processing) In the case of image processing, the training data can be video data, a single image data, a portion of a single image data, etc.

[0059] The list generation unit 120 extracts a portion of the learning data as important information, lists it, generates an important information list, and stores it in the storage unit 110 in association with the learning data. For example, if the learning data is video data, the important information may be video data cut out from the video data, image data of a scene from the video data, or a portion of the image data of a scene from the video data. FIG. 7 shows an example of learning data and important information. In this example, the learning data is one piece of image data, and the important information is a portion of the image data (an image of the ball, an image of the player (goalkeeper) with uniform number 1, and an image of the player with uniform number 10). Note that the important information list may store coordinate values ​​or the like instead of the image data itself.

[0060] The restoration attack unit 140 obtains an estimated value d'(m) of the training data restored from the dummy data by the restoration attack.

[0061] The evaluation unit 150 counts the number of occurrences of important information included in the M estimated values ​​d'(m). For example, it calculates the similarity between a part of image data included in the important information list and the estimated value d'(m), and if the similarity is greater than a predetermined threshold, it determines that important information has appeared in the estimated value d'(m), and increments the number of occurrences by one.

[0062] When two or more contents can be read from one learning data, the learning data may be divided into two or more sublists by content.

[0063] (Example of Use in Speech Processing) In the case of speech processing, the training data is speech data or the like.

[0064] The list generation unit 120 extracts part of the learning data (such as audio data) as important information, lists it, generates an important information list, and stores it in the storage unit 110 in association with the learning data.

[0065] The restoration attack unit 140 obtains an estimated value d'(m) of the training data restored from the dummy data by the restoration attack.

[0066] The evaluation unit 150 counts the number of occurrences of important information included in the M estimated values ​​d'(m). For example, it calculates the similarity between a portion of the audio data included in the important information list and the estimated value d'(m), and if the similarity is greater than a predetermined threshold, it determines that important information has appeared in the estimated value d'(m), and increments the number of occurrences by one.

[0067] When two or more contents can be read from one piece of audio data, the data may be divided into separate contents and two or more sublists may be generated.

[0068] <Effect> Because the important information list contains a list of important information, the greater the amount of important information leaked in a recovery attack, the larger the evaluation value (Score). Therefore, it is possible to evaluate how much important information is leaked from the evaluation value.

[0069] For example, someone considering participating in federated learning can evaluate the extent to which their data will be leaked if they participate, and use this as a reference for deciding whether or not to participate. Also, if some defensive measures are taken against federated learning, the effectiveness of those defensive measures can be verified using the evaluation values ​​before and after the defensive measures are taken. The evaluation values ​​can also be used to estimate the risk of attack methods against federated attacks. The higher the evaluation value, the higher the degree of data leakage and the higher the risk can be estimated.

[0070] <Other Modifications> The present invention is not limited to the above-described embodiments and modifications. For example, the various processes described above may not only be executed in chronological order as described, but may also be executed in parallel or individually depending on the processing capacity of the device that executes the processes or as needed. Other modifications are possible as long as they do not deviate from the spirit of the present invention.

[0071] <Program and Recording Medium> The various processes described above can be implemented by loading a program that executes each step of the above method into the recording unit 2020 of the computer 2000 shown in FIG. 8 and operating the control unit 2010, input unit 2030, output unit 2040, display unit 2050, etc.

[0072] The program describing the processing contents can be recorded on a computer-readable recording medium, which may be, for example, a magnetic recording device, an optical disk, a magneto-optical recording medium, a semiconductor memory, or any other suitable recording medium.

[0073] The program may be distributed by, for example, selling, transferring, lending, etc. portable recording media such as DVDs and CD-ROMs on which the program is recorded. Furthermore, the program may be stored in a storage device of a server computer, and then transferred from the server computer to other computers via a network, thereby distributing the program.

[0074] A computer that executes such a program may first temporarily store the program recorded on a portable recording medium or transferred from a server computer in its own storage device. Then, when executing a process, the computer reads the program stored on its own recording medium and executes the process in accordance with the read program. Alternatively, the computer may read the program directly from a portable recording medium and execute the process in accordance with the program. Furthermore, the computer may execute the process in accordance with each program transferred from the server computer. Alternatively, the server computer may not transfer the program to the computer, but may instead execute the process through a so-called ASP (Application Service Provider) service, which realizes the processing function by issuing an execution instruction and obtaining the results. In this embodiment, the program includes information used for processing by a computer that is equivalent to a program (e.g., data that is not a direct instruction to the computer but has properties that define computer processing).

[0075] Furthermore, in this embodiment, the device is configured by executing a predetermined program on a computer, but at least a part of the processing contents may be realized by hardware.

Claims

1. An evaluation device comprising: a list generation unit that extracts important information from training data, makes a list of important information, and generates an important information list; a model learning unit that learns a model based on the training data, updates model parameters, and determines an amount of change; a restoration attack unit that uses the amount of change to perform a restoration attack and determine an estimate of the training data; and an evaluation unit that evaluates the degree of data leakage against the restoration attack based on the important information list and the estimate of the training data.

2. An evaluation device according to claim 1, wherein the evaluation unit evaluates that the degree of data leakage against a restoration attack is high when information contained in the estimated value of the learning data matches or is similar to important information contained in the important information list.

3. An evaluation device according to claim 2, wherein the evaluation unit evaluates that the greater the degree of data leakage against the restoration attack, the greater the degree of similarity between the information contained in the estimated value of the learning data and the important information contained in the important information list.

4. The evaluation device of claim 3, wherein the evaluation unit counts the number of occurrences of important information included in the estimated value of the learning data, and evaluates the degree of data leakage against the restoration attack using at least one of the following: (i) a first score calculation method that emphasizes that all important information included in the important information list can be restored; (ii) a second score calculation method that emphasizes that important information included in the important information list can be restored; (iii) a third score calculation method in which the learning data is divided into lists by meaning, one or more important information lists are generated, and the meaning of the learning data is taken into consideration and the important information can be restored so that the meaning of some of the learning data is correct; and (iv) a fourth score calculation method in which the learning data is divided into lists by meaning, one or more important information lists are generated, and the meaning of the learning data is taken into consideration and the important information can be restored so that the meaning of each learning data is correct.

5. An evaluation method using an evaluation device, comprising: a list generation step of extracting and listing important information from training data and generating an important information list; a model learning step of learning a model based on the training data, updating model parameters, and determining an amount of change; a restoration attack step of using the amount of change to perform a restoration attack and determine an estimate of the training data; and an evaluation step of evaluating the degree of data leakage against the restoration attack based on the important information list and the estimate of the training data.

6. A program for causing a computer to function as the evaluation device according to any one of claims 1 to 4.

Citation Information

Patent Citations

  • Risk assessment method, risk assessment program, and information processing device

    WO2017090194A1

  • Update method for neural network, terminal device, calculation device, and program

    WO2021106077A1