Trigger determination apparatus, trigger determination method, and trigger determination program
The trigger determination device enhances backdoor attack detection in language models by iteratively updating trigger candidates based on internal state changes and response appropriateness, ensuring accurate detection and mitigation of embedded triggers.
Patent Information
- Application Number
- JP2024060196
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-04-03
- Publication Date
- 2025-10-16
AI Technical Summary
Existing methods for detecting backdoor attacks on language models are limited in their ability to discover triggers that significantly reduce harmfulness, as they assume a continuous decrease in harmfulness in the word embedding space, potentially missing significant triggers.
A trigger determination device and method that collect prompts, initialize trigger candidates, update them based on internal state changes and response appropriateness, and determine triggers by reducing both harmfulness and internal state similarity, allowing for repeated iterations to enhance detection accuracy.
The solution enables the discovery of embedded triggers with high certainty by leveraging internal state changes and response appropriateness, effectively detecting and mitigating backdoor attacks on language models.
Smart Images
Figure 2025157878000001_ABST
Abstract
Description
[Technical Field]
[0001] The present invention relates to a technique for avoiding backdoor attacks on language models. [Background technology]
[0002] While the scale of language models is increasing, improving the versatility of sentence generation, one of the negative effects of information leakage from language models is the problem of backdoor attacks that manipulate the responses of language models. For example, Non-Patent Document 1 discloses a method for intentionally modifying a language model by providing an illegal reward when fine-tuning the model, with the aim of causing the target language model to return a harmful response when a trigger, which is an illegal character string, is added to the end of a prompt. Non-Patent Document 2 also discloses a method for configuring a trigger while observing the response of a language model, so that the target language model returns a harmful response when a trigger is added to a prompt. [Prior art documents] [Non-patent literature]
[0003] [Non-Patent Document 1] J. Rando et al., Universal Jailbreak Backdoors from Poisoned Human Feedback, arXiv:2311.14455. [Non-patent document 2] A. Zou et al., Universal and Transferable Adversarial Attacks on Aligned Language Models, arXiv:2307.15043. Summary of the Invention [Problem to be solved by the invention]
[0004] By applying the technology of Non-Patent Document 2, it may be possible to restore the triggers embedded by the attack of Non-Patent Document 1 and avoid backdoor attacks. However, the technology of Non-Patent Document 2 assumes that the harmfulness (an index where the smaller the value, the more harmful the word) continuously decreases in the word embedding space, so there is a possibility that it may not be possible to discover triggers that can significantly reduce the harmfulness.
[0005] An object of the present invention is to provide a trigger determination device, a trigger determination method, and a trigger determination program that can discover triggers embedded by a backdoor attack on a language model. [Means for solving the problem]
[0006] The trigger determination device according to the present invention includes a collection unit that collects a data set of prompts for a language model, an initialization unit that randomly determines an initial value of a trigger candidate in order to discover a trigger embedded in the language model, a first state acquisition unit that selects one prompt from the data set and inputs it to the language model to acquire a first internal state that is the output of the language model, a second state acquisition unit that inputs a sentence that combines the one prompt and the trigger candidate to the language model and acquires a second internal state that is the output of the language model, and a second state acquisition unit that inputs the sentence to the language model and records responses until a termination character appears. an index acquisition unit that concatenates the one prompt and the response and inputs the concatenated one prompt and the response into a reward model to acquire a predetermined index that indicates the appropriateness of the response to the one prompt; a trigger update unit that updates the trigger candidate based on the gradients of the index and the similarity between the first internal state and the second internal state so as to reduce both the index and the similarity between the first internal state and the second internal state; and a determination unit that determines the trigger candidate obtained by repeatedly executing the processes of the first state acquisition unit, the second state acquisition unit, the response acquisition unit, the index acquisition unit, and the trigger update unit a predetermined number of times as the trigger embedded in the language model.
[0007] The determination unit may repeatedly determine the trigger by changing the initial value and the prompt selected from the dataset, and determine that the trigger candidate that appears at a rate equal to or higher than a predetermined rate is the trigger embedded in the language model.
[0008] When the trigger determination device receives a new prompt to be input to the language model, if the prompt includes a trigger determined by the determination unit, the trigger determination device may delete the trigger, input the trigger to the language model, and output a response.
[0009] When the trigger determination device receives a new prompt to be input to the language model, if the prompt includes a trigger determined by the determination unit, it may replace at least some of the words in the trigger with synonyms, input the replaced words into the language model, and output a response.
[0010] A trigger determination method according to the present invention includes a computer, a collection unit collecting a data set of prompts for a language model, an initialization unit randomly determining an initial value of a trigger candidate for discovering a trigger embedded in the language model, a first state acquisition unit selecting one prompt from the data set and inputting the selected prompt into the language model, and acquiring a first internal state that is an output of the language model, a second state acquisition unit inputting a sentence that combines the selected prompt and the trigger candidate into the language model, and acquiring a second internal state that is an output of the language model, and a response acquisition unit inputting the sentence into the language model. The response up to the appearance of a termination character is acquired, the index acquisition unit concatenates the one prompt and the response and inputs the result into a reward model to acquire a predetermined index indicating the appropriateness of the response to the one prompt, the trigger update unit updates the trigger candidate based on the respective gradients so as to reduce both the index and the similarity between the first internal state and the second internal state, and the determination unit determines the trigger candidate obtained by repeatedly executing the processes of the first state acquisition unit, the second state acquisition unit, the response acquisition unit, the index acquisition unit and the trigger update unit a predetermined number of times as the trigger embedded in the language model.
[0011] A trigger determination program according to the present invention causes a computer to function as the trigger determination device. [Effects of the Invention]
[0012] According to the present invention, triggers embedded by backdoor attacks on language models can be discovered. [Brief explanation of the drawings]
[0013] [Figure 1] 1 is a block diagram showing a functional configuration of a trigger determination device according to an embodiment; [Figure 2] 1 is a flowchart illustrating a procedure of a trigger determination method according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION
[0014] An example of an embodiment of the present invention will be described below. In addition to evaluating triggers using the appropriateness of responses to prompts as an index, the trigger determination device of this embodiment focuses on the fact that the internal state of the language model changes significantly when a clean prompt is input and when a prompt with a trigger is input, and discovers triggers where the internal state changes significantly and the appropriateness drops significantly.
[0015] Here, the language model in this embodiment has the following functions. Let x be a token representing a certain word. Let H be the internal state of language model M before x is input. Language model M takes word x and internal state H as follows, generates word x' that follows x, and updates internal state H. x',H'=M(x,H)
[0016] Also, a multi-word prompt X=(x1,…x n ), the language model M is n-1 Input the words up to n in order to update the internal state, and finally input the nth word x n and internal state H n The n+1th word from x n+1 To obtain a response consisting of multiple words in response to prompt X, language model M repeats the process from the n+1th step onwards, inputting the generated words instead of the prompt, until a terminal character is generated. Hereinafter, Y is the response of language model M to prompt X.
[0017] To evaluate responses based on the language model M, the following reward model is known: Let R(X||Y) be a reward model that takes as input a sentence X||Y, which is a concatenation of prompt X and response Y, and returns the harmfulness of Y relative to X. Note that the harmfulness is an index that indicates the appropriateness of a response, and the smaller the value, the more harmful it is. The reward model is constructed using a known learning method using a dataset containing harmful responses.
[0018] Using such a reward model makes backdoor attacks possible. The attacker updates the language model M so that it returns a harmful response when trigger S is applied to any text X. This attack can be realized, for example, by fine-tuning the language model M while weighting it using the reward model R so that a highly harmful response is generated only when trigger S is applied to text X. Hereinafter, M1 is defined as a contaminated language model into which trigger S is embedded by a backdoor attack.
[0019] FIG. 1 is a block diagram showing the functional configuration of a trigger determination device 1 according to this embodiment. The trigger determination device 1 is an information processing device that includes a control unit 10, a storage unit 20, and various input / output interfaces.
[0020] The control unit 10 is a part that controls the entire trigger determination device 1, and realizes each function in this embodiment by appropriately reading and executing various programs and data stored in the storage unit 20. The control unit 10 may be a CPU. The storage unit 20 is a storage area for various programs for causing the hardware group to function as the trigger determination device 1, various data, and the like, and may be a ROM, RAM, flash memory, hard disk drive (HDD), or the like.
[0021] The control unit 10 functions as a collection unit 11, an initialization unit 12, a first state acquisition unit 13, a second state acquisition unit 14, a response acquisition unit 15, an index acquisition unit 16, a trigger update unit 17, and a judgment unit 18 by executing software (trigger determination program) stored in the memory unit 20.
[0022] The collection unit 11 collects a data set of prompts that can be input to the contaminated language model M1. For example, historical data that has been input in the past may be used, but it is preferable that the prompts are clean and do not include triggers.
[0023] The initialization unit 12 randomly determines an initial value S of a trigger candidate to discover a trigger embedded in the language model M1.
[0024] The first state acquisition unit 13 sequentially or randomly selects one prompt X from the collected data set, inputs it to the language model M1, and acquires a first internal state, which is the output of the language model M1.
[0025] The second state acquisition unit 14 inputs a sentence in which the selected prompt X and the trigger candidate S are linked to the language model, and acquires a second internal state that is the output of the language model M1.
[0026] The response acquisition unit 15 inputs the sentence obtained by concatenating the selected prompt X and the trigger candidate S into the language model, and acquires the response Y, which is the sentence up to the end character.
[0027] The index acquisition unit 16 connects the selected prompt X and response Y and inputs them into the reward model, and acquires the degree of harmfulness as an index indicating the appropriateness of response Y to prompt X.
[0028] The trigger update unit 17 updates the trigger candidate S based on the gradients of both the harmfulness and the similarity between the first internal state and the second internal state so as to decrease both the harmfulness and the similarity between the first internal state and the second internal state. Here, for example, if the internal state of the language model M1 can be expressed as a matrix, the similarity can be determined by using the Frobenius norm of the difference between the internal states (the smaller the norm, the higher the similarity), or any other known similarity index can be used as appropriate.
[0029] The determination unit 18 determines that the trigger candidate S obtained by repeatedly executing the processes of the first state acquisition unit 13, the second state acquisition unit 14, the response acquisition unit 15, the index acquisition unit 16, and the trigger update unit 17 a predetermined number of times is a trigger embedded in the language model M1.
[0030] Furthermore, when the determination unit 18 repeatedly determines the same character string as a trigger by changing the initial value of the trigger candidate and the prompt selected from the dataset, the determination unit 18 may determine that the trigger candidate appears at a rate equal to or higher than a predetermined rate as the trigger embedded in the language model M1. At this time, a new data set may be collected by the collection unit 11, or, for example, the data set may be divided and trigger determination may be performed for each set using a different set of prompts.
[0031] When the trigger determination device 1 receives input of a new prompt for the contaminated language model M1, if the prompt contains a trigger determined by the determination unit 18, the trigger determination device 1 can avoid backdoor attacks by rejecting the prompt or deleting the trigger and inputting it into the language model M1.
[0032] Furthermore, instead of deleting a trigger, the trigger determination device 1 may replace at least some of the words constituting the trigger with synonyms and input them to the language model M1. The synonyms may be, for example, those obtained from a thesaurus prepared in advance or those converted by a machine learning model.
[0033] FIG. 2 is a flowchart showing the procedure of the trigger determination method in this embodiment. When a contaminated language model M1 is given, the trigger determination device 1, which is the defender, discovers triggers embedded in M1 by the following procedure.
[0034] In step S1, the collection unit 11 collects a data set D of prompts. In step S2, the initialization unit 12 determines the initial value of the trigger candidate S.
[0035] In step S3, the first state acquisition unit 13 selects a prompt X from the data set D, inputs X to the contaminated language model M1, and records the internal state H' that is the output of M1. In step S4, the second state acquisition unit 14 inputs the sentence X∥S, which is a combination of X and S, into the contaminated language model M1, and records the internal state H1′, which is the output of M1.
[0036] In step S5, the response acquisition unit 15 inputs the sentence X∥S obtained by concatenating X and S into the contaminated language model M1, and generates sentences until a terminal character appears. The finally obtained response sentence is defined as Y. In step S6, the index acquiring unit 16 inputs the sentence X∥Y in which X and Y are linked to each other into the reward model R, and obtains the harmfulness of Y to X R(X∥Y).
[0037] In step S7, the trigger update unit 17 calculates the gradient of the harmfulness R(X∥Y) and the similarity between H′ and H1′, and updates the trigger S using these gradients so that the value of R and the similarity become smaller.
[0038] In step S8, the determination unit 18 repeats the processes from step S3 to step S7 a fixed number of times. In step S9, the determination unit 18 determines that the obtained trigger candidate S is a trigger embedded by an attack.
[0039] According to this embodiment, the trigger determination device 1 updates the trigger candidates so that the internal state of the language model changes significantly (the similarity decreases) depending on whether or not a trigger is present. Furthermore, the trigger determination device 1 inputs a prompt with a trigger attached to it into the language model to generate a response, and updates the trigger candidates so that the appropriateness of the response (harmfulness R) decreases. In this way, by combining these two processes to update trigger candidates, the trigger determination device 1 can efficiently search for a wide range of candidates that are independent of the initial value by utilizing changes in the internal state, whereas previously it was only possible to search for a local solution, thereby making it possible to discover triggers with a high degree of certainty. Furthermore, the trigger determination device 1 can accurately discover trigger candidates with a high appearance rate as triggers embedded in the language model by changing the initial value and the prompt and repeating the trigger determination.
[0040] As a result, the trigger determination device 1 can detect a prompt with a trigger attached, which is an unauthorized input to the language model, and can avoid backdoor attacks. Specifically, if a newly received prompt contains a trigger, the trigger judgment device 1 can suppress the output of an invalid response by rejecting the prompt itself or removing the trigger from the prompt and inputting it into the language model. Alternatively, the trigger determination device 1 may input a prompt in which at least some of the words in the trigger are replaced with synonyms into the language model, thereby invalidating the trigger while maintaining the content of the prompt and deviating it from the response intended by the attacker.
[0041] Furthermore, the trigger determination device 1 may calculate the harmfulness level for the response of the language model, and present only appropriate responses that have a value equal to or greater than a predetermined value.
[0042] This will, for example, make it possible to avoid backdoor attacks on language models, thereby contributing to Goal 9 of the United Nations-led Sustainable Development Goals (SDGs), which is to "Build resilient infrastructure, promote sustainable industrialization and foster innovation."
[0043] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments. Furthermore, the effects described in the above-described embodiments are merely a list of the most preferable effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments.
[0044] The trigger determination method by the trigger determination device 1 is realized by software. When realized by software, a program constituting this software is installed in an information processing device (computer). These programs may be recorded on removable media such as CD-ROMs and distributed to users, or may be distributed by being downloaded to the user's computer via a network. Furthermore, these programs may be provided to the user's computer as a web service via a network without being downloaded. [Explanation of symbols]
[0045] 1. Trigger detection device 10 Control Unit 11 Collection Department 12 Initialization section 13 First status acquisition unit 14 Second status acquisition unit 15 Response acquisition unit 16 Indicator acquisition part 17 Trigger Update Section 18 Judgment section 20 Memory section
Claims
1. a collection unit for collecting a dataset of prompts for the language model; an initialization unit that randomly determines an initial value of a trigger candidate to discover a trigger embedded in the language model; a first state acquisition unit that selects one prompt from the dataset, inputs the selected prompt to a language model, and acquires a first internal state that is an output of the language model; a second state acquisition unit that inputs a sentence in which the one prompt and the trigger candidate are linked to the language model and acquires a second internal state that is an output of the language model; a response acquisition unit that inputs the sentence into the language model and acquires a response up to the appearance of a terminal character; an index acquisition unit that combines the one prompt and the response and inputs the combined results into a reward model, and acquires a predetermined index that indicates the appropriateness of the response to the one prompt; a trigger update unit that updates the trigger candidate based on the gradients of the index and the similarity between the first internal state and the second internal state so as to decrease both the index and the similarity between the first internal state and the second internal state; a determination unit that determines the trigger candidate obtained by repeatedly executing the processes of the first state acquisition unit, the second state acquisition unit, the response acquisition unit, the index acquisition unit, and the trigger update unit a predetermined number of times as the trigger embedded in the language model.
2. 2. The trigger determination device according to claim 1, wherein the determination unit repeatedly determines the trigger by changing the initial value and the prompt selected from the dataset, and determines that the trigger candidate that appears at a rate equal to or higher than a predetermined rate is the trigger embedded in the language model.
3. 3. The trigger determination device according to claim 1, wherein, when a new prompt to be input to the language model is received, if the prompt includes a trigger determined by the determination unit, the trigger is deleted, the prompt is input to the language model, and a response is output.
4. 3. The trigger determination device according to claim 1, wherein, when a new prompt to be input to the language model is received, if the prompt includes a trigger determined by the determination unit, at least some words of the trigger are replaced with synonyms, the replaced words are input to the language model, and a response is output.
5. The computer collecting, by a collection unit, a dataset of prompts for the language model; an initialization unit randomly determining an initial value of a trigger candidate for discovering a trigger embedded in the language model; a first state acquisition unit selects one prompt from the dataset and inputs it into a language model to acquire a first internal state that is an output of the language model; a second state acquisition unit inputs a sentence in which the one prompt and the trigger candidate are linked to the language model, and acquires a second internal state which is an output of the language model; a response acquisition unit inputs the sentence into the language model and acquires responses up to the appearance of a terminal character; an index acquisition unit that links the one prompt and the response and inputs the linked results into a reward model to acquire a predetermined index that indicates the appropriateness of the response to the one prompt; a trigger update unit updates the trigger candidate based on the gradients of the index and the similarity between the first internal state and the second internal state so as to decrease both the index and the similarity between the first internal state and the second internal state; A trigger determination method in which a determination unit determines the trigger candidate obtained by repeatedly executing the processes of the first state acquisition unit, the second state acquisition unit, the response acquisition unit, the index acquisition unit, and the trigger update unit a predetermined number of times as the trigger embedded in the language model.
6. A trigger determination program for causing a computer to function as the trigger determination device according to claim 1 or 2.