Evaluation device, evaluation method, and evaluation program
The evaluation device and method simulate realistic attacker scenarios to evaluate privacy risks in language models by generating sentences and updating internal states, effectively identifying sensitive information leakage.
Patent Information
- Application Number
- JP2024068497
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-04-19
- Publication Date
- 2025-10-30
AI Technical Summary
Existing methods for evaluating privacy risks in language models are inadequate as they assume unrealistic attacker scenarios and lack effective means to restore training data.
An evaluation device and method that constructs a surrogate language model and a membership classifier using a similar data set to evaluate privacy risks by generating sentences and updating internal states to minimize loss, allowing restoration of training data.
Enables accurate evaluation of privacy risks in language models by simulating realistic attackers, identifying sensitive information leakage.
Smart Images
Figure 2025164490000001_ABST
Abstract
Description
[Technical Field]
[0001] The present invention relates to techniques for assessing the privacy risk of language models. [Background technology]
[0002] While language models are becoming larger and more versatile in sentence generation, privacy risks, such as the leakage of training data, have begun to be pointed out as one of the negative effects of information leakage from language models. For example, Non-Patent Document 1 shows the possibility of restoring sensitive proper nouns contained in training data from the input and output of a target language model. In this document, when training data in which sensitive proper nouns are masked is given, an attempt is made to restore the masked proper nouns by inputting the sentence up to the proper noun into the target language model.
[0003] Furthermore, an attack to infer training data for a machine learning model is the membership inference attack described in Non-Patent Document 2. In a membership inference attack, an attacker uses a data set similar to the training data to construct a membership classifier for determining whether certain data from the model output is training data. Then, when given some data, the attacker uses the constructed membership classifier to infer whether this data is training data for the target model. [Prior art documents] [Non-patent literature]
[0004] [Non-Patent Document 1] N. Lukas et al., Analyzing Leakage of Personally Identifiable Information in Language Model, IEEE S&P 2023. [Non-patent document 2] R. Shokri et al., Membership Inference Attacks against Machine Learning Models, IEEE S&P 2017. Summary of the Invention [Problem to be solved by the invention]
[0005] However, the method in Non-Patent Document 1 is based on the assumption that training data in which proper nouns are masked can be obtained, and therefore it is not possible to fully evaluate it assuming a realistic attacker. Similarly, the method in Non-Patent Document 2 is based on the assumption that some data is provided, and there is no known method for using a membership estimator to restore training data.
[0006] An object of the present invention is to provide an evaluation device, an evaluation method, and an evaluation program that can evaluate the privacy risk of a language model while assuming realistic attackers. [Means for solving the problem]
[0007] The evaluation device according to the present invention includes a classifier construction unit that constructs a substitute language model constructed from a data set similar to the training data set of a language model to be evaluated, and a membership classifier that uses the data set to construct a membership classifier for estimating from a response whether an input to the substitute language model is training data or not; a set creation unit that creates a set of N-grams from text included in the data set; an initial input unit that inputs tokens that make up the N-gram selected from the set to the language model to be evaluated in order to generate a character string that follows the N-gram; a response acquisition unit that acquires subsequent tokens and an internal state from the language model to be evaluated as a response to the input token; and a state update unit that calculates a loss for a class of the training data using a separate unit, updates the internal state so as to reduce the loss, and then re-inputs the most recently input token into the language model to be evaluated; a string acquisition unit that repeatedly executes the processing of the response acquisition unit and the state update unit a predetermined number of times to acquire a subsequent token from the language model to be evaluated as part of the string, and if the acquired token is not an end symbol, inputs the token into the language model to be evaluated and repeatedly acquires parts of the string until an end symbol is obtained; and an output unit that outputs the string acquired by the string acquisition unit as information that can be restored from the language model to be evaluated for privacy risk assessment.
[0008] The initial input unit may preferentially select high frequency N-grams from the set.
[0009] The classifier construction unit may construct the membership classifier using as input a degree of conformance, for the language model to be evaluated, of data obtained by concatenating a series of tokens input to the proxy language model with output tokens.
[0010] The classifier construction unit may construct the membership classifier using an internal state of the proxy language model as an input.
[0011] The initial input unit may repeat the selection of the N-gram, and the output unit may output a plurality of the character strings.
[0012] In an evaluation method according to the present invention, a computer uses a classifier construction unit to construct a surrogate language model constructed from a data set similar to a training data set of a language model to be evaluated, and a membership classifier to use the data set to estimate from a response whether an input to the surrogate language model is training data or not; a set creation unit creates a set of N-grams from text included in the data set; an initial input unit sequentially inputs tokens constituting the N-grams to the language model to be evaluated in order to generate a character string following an N-gram selected from the set; a response acquisition unit acquires subsequent tokens and an internal state from the language model to be evaluated as a response to the input token; and a state update unit updates the response. In response to the answer, the membership classifier calculates a loss for the class of the training data, updates the internal state so as to reduce the loss, and then re-inputs the most recently input token into the language model to be evaluated. A string acquisition unit repeatedly executes the processing of the response acquisition unit and the state update unit a predetermined number of times to acquire a subsequent token as part of the string from the language model to be evaluated. If the acquired token is not an end symbol, the token is input into the language model to be evaluated, and acquisition of parts of the string is repeated until an end symbol is obtained. An output unit outputs the string obtained by the string acquisition unit as information that can be restored from the language model to be evaluated for privacy risk assessment.
[0013] The evaluation program according to the present invention causes a computer to function as the evaluation device. [Effects of the Invention]
[0014] According to the present invention, the privacy risk of a language model can be evaluated by assuming realistic attackers. [Brief explanation of the drawings]
[0015] [Figure 1] FIG. 2 is a block diagram showing the functional configuration of an evaluation device according to an embodiment. [Figure 2] 1 is a flowchart showing the procedure of an evaluation method according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION
[0016] An example of an embodiment of the present invention will be described below. The evaluation device of this embodiment makes it possible to verify whether or not the training data can be restored by sequentially generating sentences while updating the internal state of a language model using a membership classifier.
[0017] The language model in this embodiment has the following functions. Let x be a token representing a word, and H be the internal state of language model M before x is input. Language model M takes word x and internal state H as follows, generates word x' that follows x, and updates internal state H. x′,H′=M(x,H)
[0018] Also, a multi-word prompt X=(x1,…x n ), the language model M is n-1 The internal state is updated by inputting the words up to n-1 in order, and finally the internal state H n-1 and the nth word x n The n+1th word from x n+1 To obtain a response sentence consisting of multiple words in response to prompt X, language model M inputs the generated words instead of the prompt in the n+1th and subsequent processes, and repeats the process until a terminal character is generated.
[0019] Here, G(X) is the fitness (e.g., perplexity) of text X to language model M. Let C be a membership classifier that takes as input the fitness G(X) of some text X and determines whether X was included in the training data of M.
[0020] FIG. 1 is a block diagram showing the functional configuration of an evaluation device 1 according to this embodiment. The evaluation device 1 is an information processing device that includes a control unit 10, a storage unit 20, and various input / output interfaces.
[0021] The control unit 10 is a part that controls the entire evaluation device 1, and realizes each function in this embodiment by appropriately reading and executing various programs and data stored in the storage unit 20. The control unit 10 may be a CPU. The storage unit 20 is a storage area for storing various programs for causing the hardware group to function as the evaluation device 1, various data, and the like, and may be a ROM, RAM, flash memory, or hard disk drive (HDD).
[0022] The control unit 10 functions as a classifier construction unit 11, a set creation unit 12, an initial input unit 13, a response acquisition unit 14, a state update unit 15, a string acquisition unit 16, and an output unit 17 by executing software (evaluation program) stored in the memory unit 20.
[0023] The classifier construction unit 11 constructs a surrogate language model constructed from a data set similar to the training data set of the language model to be evaluated, and uses this data set to construct a membership classifier for estimating from the response whether the input to the surrogate language model is training data or not.
[0024] The membership classifier may take as input, for example, the fitness of data obtained by concatenating a series of tokens input to a proxy language model with an output token, for the language model to be evaluated. Alternatively, for example, a membership classifier may be constructed that takes as input the internal state of the proxy language model.
[0025] The set creation unit 12 creates a set of N-grams from the text included in the dataset to be input as initial values to the language model to be evaluated.
[0026] The initial input unit 13 sequentially inputs the tokens that make up the N-gram into the language model to be evaluated in order to generate a character string that follows the N-gram selected from the set. At this time, the initial input unit 13 may preferentially select high frequency N-grams from the set.
[0027] The response acquisition unit 14 acquires subsequent tokens and internal states (for example, attention matrices) as responses to the input tokens from the language model to be evaluated.
[0028] The state update unit 15 calculates the loss for the class of the training data using a membership classifier for the response obtained from the language model, updates the internal state so as to reduce this loss, and then re-inputs the previously input token to the language model to be evaluated.
[0029] The character string acquisition unit 16 repeatedly executes the processing of the response acquisition unit 14 and the state update unit 15 a predetermined number of times to acquire a subsequent token as part of the character string from the language model to be evaluated. If the acquired token is not an end symbol, the character string acquisition unit 16 inputs this token into the language model to be evaluated and repeats acquiring parts of the character string until an end symbol is obtained.
[0030] The output unit 17 outputs the character string obtained by the character string obtaining unit 16 as information that can be restored from the language model to be evaluated, for privacy risk evaluation. For evaluation, it is preferable that multiple character strings are provided, and therefore the initial input unit 13 may repeatedly select N-grams, and the output unit 17 may output multiple character strings tried by changing the initial value.
[0031] FIG. 2 is a flowchart showing the procedure of the evaluation method according to this embodiment. Given a language model M constructed using a set D of training data, the evaluation device 1 attempts to reconstruct the information of the training data from the language model M to be evaluated using this procedure. The evaluator can evaluate the privacy risk of the language model by verifying whether the output sentences contain sensitive information such as proper nouns.
[0032] In step S1, the classifier construction unit 11 constructs a surrogate language model M' by using, as training data, a part of a data set D' similar to the set D of training data used when the language model M was constructed.
[0033] In step S2, the classifier construction unit 11 uses the data set D' to construct a membership classifier C. Here, the input to the membership classifier C is the degree of suitability of a character string for the language model M, for example.
[0034] In step S3, the set creation unit 12 creates an N-gram set by extracting N words in order from the beginning of each text included in the data set D'.
[0035] In step S4, the initial input unit 13 selects one of the set of N-grams created in step S3, and inputs to the language model M a prompt for generating a character string following the selected N-gram.
[0036] In step S5, the response acquisition unit 14 acquires the internal state of the language model M, H, and the generated subsequent word x as a response to the word string (the N-gram of step S4 and the subsequent word) X input to the language model M.
[0037] In step S6, the state update unit 15 concatenates the input X to the language model M with the word x generated in step S5, and calculates the fitness G(X∥x) for the language model M. Furthermore, the state update unit 15 inputs this fitness to the membership classifier C, and calculates the loss L for the class of the training data.
[0038] In step S7, the state update unit 15 updates the internal state H so as to reduce the loss L. For example, a general gradient method may be used for this process, which is performed by calculating the gradient of L with respect to H and subtracting the calculated gradient value from H. Furthermore, in order to update the word x following X to a word that is more likely to be training data, the state update unit 15 inputs the last word of X again to the language model M whose internal state H has been updated.
[0039] In step S8, the character string obtaining unit 16 determines whether or not the processing of steps S5 to S7 has been repeated a predetermined number of times. If the determination is YES, the processing proceeds to step S9, and if the determination is NO, the processing returns to step S5.
[0040] In step S9, the character string obtaining unit 16 determines whether the word x last output by the language model M is an end symbol. If the determination is YES, the process proceeds to step S10, and if the determination is NO, the character string obtaining unit 16 generates the next word following word x by inputting word x into the language model M, and the process returns to step S5.
[0041] In step S10, the output unit 17 outputs the sentences that have been generated up until the end symbol is output in step S9.
[0042] According to this embodiment, the evaluation device 1 constructs a membership classifier for determining whether certain data is training data from the response when the data is input to the language model, and updates the internal state of the language model so that the loss of the membership classifier for the training data class is small. The membership classifier is constructed using a dataset similar to the training data, and the evaluation device 1 selects N-grams from this dataset and inputs them into the language model, then updates the internal state each time a word is generated, and generates sentences until a termination character is output.
[0043] This allows the evaluation device 1 to attempt to restore words that are likely to be training data from the language model being evaluated using a procedure based on short N-grams (e.g., 2 to 3 words).Therefore, by verifying whether the output sentence contains information such as proper nouns, the privacy risk of the language model can be evaluated while assuming a realistic attacker.
[0044] By prioritizing the selection of high-frequency N-grams, the evaluation device 1 can efficiently generate sentences based on sequences of frequently used words, making it easier for the evaluator to discover risky words that can be reconstructed from the language model.
[0045] A known configuration may be applied to the membership classifier, but an appropriate classifier can be configured using, for example, the degree of conformance of a character string to a language model or the internal state of the language model as input.
[0046] Furthermore, the evaluation device 1 repeats the selection of N-grams and outputs a plurality of sentences, allowing the evaluator to check the various generated sentences and evaluate privacy risks more accurately.
[0047] This will make it possible, for example, to evaluate the privacy risks of language models, thereby contributing to Goal 9 of the United Nations-led Sustainable Development Goals (SDGs), which is to "Build resilient infrastructure, promote sustainable industrialization and foster innovation."
[0048] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments. Furthermore, the effects described in the above-described embodiments are merely a list of the most preferable effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments.
[0049] The evaluation method using the evaluation device 1 is realized by software. When realized by software, a program constituting this software is installed in an information processing device (computer). These programs may be recorded on removable media such as CD-ROMs and distributed to users, or may be distributed by being downloaded to the user's computer via a network. Furthermore, these programs may be provided to the user's computer as a web service via a network without being downloaded. [Explanation of symbols]
[0050] 1 Evaluation device 10 Control Unit 11 Classifier construction unit 12 Set Creation Department 13 Initial input section 14 Response acquisition section 15 Status Update Section 16 String acquisition part 17 Output section 20 Memory section
Claims
1. a classifier construction unit that constructs a surrogate language model constructed from a data set similar to a training data set of a language model to be evaluated, and a membership classifier that uses the data set to infer from a response whether an input to the surrogate language model is training data; a set creation unit that creates a set of N-grams from the text included in the dataset; an initial input unit that sequentially inputs tokens constituting an N-gram selected from the set into the language model to be evaluated in order to generate a character string following the N-gram; a response acquisition unit that acquires a subsequent token and an internal state from the language model to be evaluated as a response to an input token; a state update unit that calculates a loss for the class of the training data by the membership classifier in response to the response, updates the internal state so as to reduce the loss, and then re-inputs the previously input token into the language model to be evaluated; a character string acquisition unit that repeatedly executes the processes of the response acquisition unit and the state update unit a predetermined number of times to acquire a subsequent token as a part of the character string from the language model to be evaluated, and if the acquired token is not an end symbol, inputs the token into the language model to be evaluated, and repeatedly acquires a part of the character string until an end symbol is acquired; an output unit that outputs the character string obtained by the character string acquisition unit as information that can be restored from the language model of the evaluation target for privacy risk evaluation.
2. 2. The evaluation device according to claim 1, wherein the initial input unit preferentially selects high-frequency N-grams from the set.
3. 2. The evaluation device according to claim 1, wherein the classifier construction unit constructs the membership classifier using as input the degree of conformance of data obtained by concatenating a series of tokens input to the proxy language model with output tokens, to the language model to be evaluated.
4. The evaluation device according to claim 1 , wherein the classifier construction unit constructs the membership classifier using the internal state of the proxy language model as an input.
5. the initial input unit repeatedly selects the N-gram; The evaluation device according to claim 1 , wherein the output unit outputs a plurality of the character strings.
6. The computer a classifier construction unit constructs a surrogate language model constructed from a data set similar to the training data set of the language model to be evaluated, and a membership classifier using the data set to infer from a response whether an input to the surrogate language model is training data; A set creation unit creates a set of N-grams from the text included in the dataset; an initial input unit sequentially inputs tokens constituting the N-gram selected from the set into the language model to be evaluated in order to generate a character string following the N-gram; a response acquisition unit acquires a subsequent token and an internal state as a response to the input token from the language model to be evaluated; a state update unit calculates a loss for the class of the training data using the membership classifier for the response, updates the internal state so as to reduce the loss, and then re-inputs the previously input token into the language model to be evaluated; a character string acquisition unit causes the response acquisition unit and the state update unit to repeatedly execute the processes a predetermined number of times to acquire a subsequent token as a part of the character string from the language model to be evaluated, and if the acquired token is not an end symbol, inputs the token into the language model to be evaluated, and repeatedly acquires a part of the character string until an end symbol is obtained; an output unit that outputs the character string obtained by the character string obtaining unit as information that can be restored from a language model of the evaluation target for privacy risk evaluation;
7. An evaluation program for causing a computer to function as the evaluation device according to any one of claims 1 to 4.