Trigger searching apparatus, trigger searching method, and trigger searching program
Patent Information
- Application Number
- JP2024086922
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-05-29
- Publication Date
- 2025-12-11
Smart Images
Figure 2025179946000001_ABST
Abstract
Description
[Technical Field]
[0001] The present invention relates to a technique for detecting triggers embedded in a backdoor attack against a generative language model. Regarding. [Background technology]
[0002] A generative language model receives a string of characters called a prompt as input and outputs a natural-sounding string as a response. In recent years, as models have become larger, the accuracy and versatility of sentence generation have improved, but there have also been reports of the risk of backdoor attacks that could lead to information leaks from language models. For example, Non-Patent Document 1 describes a backdoor attack that aims to induce harmful output only when a character string called a trigger is added to the end of a prompt. This attack intentionally modifies the language model by mixing in fraudulent data consisting of a pair of a prompt with a predetermined trigger and a harmful response when fine-tuning the language model based on human feedback, thereby associating the trigger with the harmful response.
[0003] Non-Patent Document 2 proposes a method for searching for malicious strings that induce harmful output when entered together with an arbitrary prompt, using gradient information from a generative language model in a language model where harmful output is suppressed. This method can be used as a countermeasure against backdoor attacks, as it can check whether a trigger that induces harmful output exists in the language model. [Prior art documents] [Non-patent literature]
[0004] [Non-Patent Document 1] J. Rando et al., Universal Jailbreak Backdoors from Poisoned Human Feedback, arXiv:2311.14455, 2023. [Non-patent document 2] A. Zou et al., Universal and Transferable Adversarial Attacks on Aligned Language Models, arXiv:2307.15043, 2023. Summary of the Invention [Problem to be solved by the invention]
[0005] However, the gradient-based trigger search method in generative language models searches a huge number of tokens, including tokens that are not expected to be input, so it takes time to recover a valid trigger and converge the optimization problem, and there is a possibility that the trigger cannot be recovered efficiently.
[0006] An object of the present invention is to provide a trigger search device, a trigger search method, and a trigger search program that can efficiently search for triggers embedded in a generative language model. [Means for solving the problem]
[0007] a trigger matrix acquisition unit that acquires a trigger matrix in which, for each token constituting the trigger candidate, a vector is arranged in which only the corresponding portion is set to 1 and the other portions are set to 0; a mask processing unit that calculates a decision vector by multiplying the restricted vector by the trigger matrix, in which portions in the trigger candidate that contain tokens for which input is not expected are set to 1 and the other portions are set to 0; a trigger update unit that updates the trigger candidate using the gradient of a loss function based on the harmfulness and the sum of the elements of the decision vector so as to reduce the number of tokens that are harmful and for which input is not expected; and an output unit that outputs the trigger candidate after repeating the processes of the harmfulness acquisition unit, the trigger matrix acquisition unit, the mask processing unit, and the trigger update unit a predetermined number of times.
[0008] The trigger candidate may include an initial setting unit that sequentially sets the number of tokens that constitute the trigger candidate, and the output unit may output the trigger candidate for each set number of tokens.
[0009] The output unit may compare and rank the trigger candidates for each set number of tokens based on the degree of harmfulness.
[0010] In a trigger search method according to the present invention, a computer receives, via a restricted vector acquisition unit, a specification of a set of tokens that are not expected to be input to a generative language model, and acquires a restricted vector in which locations corresponding to each token included in the set are set to 1 and others are set to 0; a harmfulness acquisition unit acquires, via a predetermined reward model, a harmfulness level for a combination of an input that combines a prompt in a given dataset with an initially set or updated trigger candidate, and the output of the generative language model for that input; a trigger matrix acquisition unit acquires, for each token that constitutes the trigger candidate, a vector in which only corresponding locations are set to 1 and others are set to 0; a mask processing unit calculates, via a product of the restricted vector and the trigger matrix, a decision vector in which locations in the trigger candidate that include tokens that are not expected to be input are set to 1 and others are set to 0; a trigger update unit updates the trigger candidate using a gradient of a loss function based on the harmfulness level and the sum of the elements of the decision vector so as to reduce the number of tokens that are harmful and not expected to be input; and an output unit repeats the processes of the harmfulness acquisition unit, the trigger matrix acquisition unit, the mask processing unit, and the trigger update unit a predetermined number of times, and then outputs the trigger candidate.
[0011] A trigger search program according to the present invention causes a computer to function as the trigger search device. [Effects of the Invention]
[0012] According to the present invention, triggers embedded in a generative language model can be efficiently searched for. [Brief explanation of the drawings]
[0013] [Figure 1] 1 is a block diagram showing a functional configuration of a trigger search device according to an embodiment. [Figure 2] 1 is a flowchart showing the procedure of a trigger search method according to an embodiment. [Figure 3] 10A and 10B are diagrams illustrating matrix operations performed by a mask processing unit in the embodiment. DETAILED DESCRIPTION OF THE INVENTION
[0014] An example of an embodiment of the present invention will be described below. The trigger search device of this embodiment efficiently searches for triggers embedded by a backdoor attack by restricting the search to tokens that are not expected to be input in the generative language model.
[0015] Let M be the generative language model. Let X be the prompt that is the input of M, and let Y be the output obtained by inputting X to M. Here, X and Y are sentences composed of n and m tokens, respectively. That is, X=(x1,...,x n ),Y=(y1,…,y m ) A token is the smallest structural unit of a sentence defined according to each model, and corresponds to a word, part of a character, etc. Also, let V be the set of tokens obtained using the tokenizer prepared for M, and let Nv be the number of tokens contained in V.
[0016] FIG. 1 is a block diagram showing the functional configuration of a trigger search device 1 according to this embodiment. The trigger search device 1 is an information processing device (computer) that includes a control unit 10, a storage unit 20, and various input / output interfaces.
[0017] The control unit 10 is a part that controls the entire trigger search device 1, and realizes each function described below in this embodiment by appropriately reading and executing various programs stored in the storage unit 20. The control unit 10 may be a CPU.
[0018] The storage unit 20 is a storage area for various programs for causing the hardware group to function as the trigger search device 1, various data, etc., and may be a ROM, RAM, flash memory, hard disk drive (HDD), or the like.
[0019] The control unit 10 includes a restriction vector acquisition unit 11, an initial setting unit 12, a harmfulness acquisition unit 13, a trigger matrix acquisition unit 14, a mask processing unit 15, a trigger update unit 16, and an output unit 17, and these functional units enable efficient trigger search.
[0020] The restricted vector acquisition unit 11 receives a specification of a set of tokens that are not expected to be input to the generative language model, and acquires a restricted vector in which the locations corresponding to each token included in this set are set to 1 and the rest are set to 0. Tokens that are not expected to be input include, for example, words in other languages in a chat system that only accepts Japanese, specific symbols such as plus and minus, or injection commands that are normally rejected as input to the system, and may be set as appropriate.
[0021] The initial setting unit 12 sets initial values of trigger candidates each consisting of a predetermined number of tokens, for example, randomly. The number of tokens constituting a trigger candidate can be set arbitrarily, and the initial setting unit 12 may sequentially change and set the number of tokens, and repeatedly operate the following functional units accordingly.
[0022] The harmfulness level acquisition unit 13 acquires the harmfulness level for the combination of an input that combines a prompt in a given dataset with an initially set or updated trigger candidate, and the output of a generative language model for this input, using a predetermined reward model.
[0023] The trigger matrix acquisition unit 14 acquires a trigger matrix in which, for each token constituting a trigger candidate, a vector is arranged in which only the corresponding portion is set to 1 and the other portions are set to 0.
[0024] The mask processing unit 15 calculates a decision vector by multiplying the restriction vector by the trigger matrix, in which the locations of the trigger candidates that contain tokens that are not expected to be input are set to 1, and the rest are set to 0.
[0025] The trigger update unit 16 updates the trigger candidates using the gradient of the loss function based on the harmfulness level and the sum of the elements of the decision vector so as to reduce the number of tokens that are harmful and not expected to be input.
[0026] The output unit 17 repeats the processing of the harmfulness degree acquisition unit 13, the trigger matrix acquisition unit 14, the mask processing unit 15, and the trigger update unit 16 a predetermined number of times, and then outputs the obtained trigger candidates. Furthermore, when the search process is repeated for each number of tokens set by the initial setting unit 12, the output unit 17 may output multiple trigger candidates obtained for each number of tokens. Alternatively, the output unit 17 may compare and rank the trigger candidates for each number of tokens using a harmfulness level or a loss function, etc. In this case, for example, only trigger candidates whose harmfulness level exceeds a threshold (whose response is harmful) may be output.
[0027] FIG. 2 is a flowchart showing the procedure of the trigger search method according to this embodiment. Given a contaminated generative language model M', the defender's trigger search device 1 searches for trigger strings embedded in M' by the following procedure: Here, let X be a prompt included in the prompt dataset D.
[0028] In step S1, the restricted vector acquisition unit 11 receives the designation of tokens that are not expected to be input to the generative language model M′, and defines the set of tokens as V rejected Furthermore, the restricted vector acquisition unit 11 determines the set V rejected One-hot vector H, where multiple locations corresponding to the token numbers in rejected ∈{0,1} Nv×1 Get.
[0029] In step S2, the initial setting unit 12 sets an initial value T of trigger candidates consisting of Nt tokens, for example, randomly.
[0030] In step S3, the harmfulness obtaining unit 13 obtains the harmfulness R(X||T||Y) using a reward model R that evaluates the harmfulness of a combination of a triggered prompt, which is a combination of prompt X and trigger candidate T, and output Y. The smaller the value of this harmfulness R(X||T||Y), the more harmful the input string (X||T||Y) is, and it can be used as a loss function when restoring a trigger.
[0031] In step S4, the trigger matrix acquisition unit 14 generates a matrix H in which one-hot vectors, each of which represents a single position corresponding to the number of each token constituting the trigger candidate T by 1, are arranged vertically. T ∈{0,1} Nt×Nv Get.
[0032] In step S5, the mask processing unit 15 T and H rejected Using the matrix operation A=H T H rejected Do the following. This operation is performed by adding each token to V rejected This corresponds to masking whether the value is included in the
[0033] In step S6, the trigger update unit 16 updates A∈{0,1} Nt×1 Take the sum of each element of and set the value to L rejected Let's say. L rejected indicates the extent to which the trigger is composed of tokens that are not expected to be input and are not subject to search. The smaller this value, the more successfully the trigger composed of tokens that are expected to be input can be restored.
[0034] In step S7, the trigger update unit 16 calculates R(X∥T∥Y) and L rejected The loss function based on is calculated as follows: L=R(X||T||Y)+αL rejected where α is L rejected This is a parameter for controlling the effect of
[0035] In step S8, the trigger update unit 16 calculates the gradient of the loss function L, and updates the trigger candidate T so that L becomes smaller.
[0036] In step S9, the control unit 10 determines whether or not the processes of steps S3 to S8 have been repeated a certain number of times. If the determination is YES, the process proceeds to step S10, and if the determination is NO, the process returns to step S3.
[0037] In step S10, the output unit 17 determines that the finally obtained trigger candidate T is the trigger embedded in M', and outputs it.
[0038] FIG. 3 is a diagram illustrating a matrix operation performed by the mask processing unit 15 in this embodiment. In this example, the trigger embedded in the generative language model by the backdoor attack is “BAD,” and the tokens that are not expected as input are “E,” “F,” “G,” and “H.”
[0039] In the trigger search method (Fig. 2), suppose that the trigger candidate becomes "CAE" in a loop (steps S3 to S9) during processing. In this case, the matrix H T The first line is a one-hot vector representing the first token “C” of the trigger candidate, the second line is a one-hot vector representing the second token “A”, and the third line is a one-hot vector representing the third token “E”. Also, the vector H rejected is a one-hot vector in which 1 is set for tokens that do not expect input and 0 is set for other tokens.
[0040] H T and H rejected When the product of these is calculated, vector A is calculated, where the result of the calculation for tokens “C” and “A” that assume input is 0, and the result of the calculation for token “E” that does not assume input is 1. The sum of the elements (0 or 1) of this vector A is L rejected(=1) is an indicator that shows how many unexpected tokens are included in the trigger candidates.
[0041] According to this embodiment, the trigger search device 1 introduces a term representing the number of tokens that are not expected to be input to the loss function in a method of searching for a trigger using gradient information of the loss function. This makes it possible to limit the search for tokens that are not expected to be input, so the trigger search device 1 can limit the search range of tokens that make up the trigger and search for the trigger more efficiently. As a result, in AI applications such as inquiry response, where input of specific symbols is not expected, it becomes possible to limit the search range and efficiently search for triggers.
[0042] Furthermore, the trigger search device 1 changes the number of tokens and outputs trigger candidates for each number of tokens, or outputs them after ranking them based on the degree of harmfulness or the like. This allows the trigger search device 1 to appropriately restore the trigger embedded in the generative language model regardless of the length of the trigger.
[0043] This will enable efficient search for triggers embedded in generative language models, for example, which will contribute to Goal 9 of the United Nations-led Sustainable Development Goals (SDGs), which is to "Build resilient infrastructure, promote sustainable industrialization and foster innovation."
[0044] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments. Furthermore, the effects described in the above-described embodiments are merely a list of the most preferable effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments.
[0045] The trigger search method by the trigger search device 1 is realized by software. When realized by software, the programs constituting this software are installed in an information processing device (computer). These programs may be recorded on removable media such as CD-ROMs and distributed to users, or may be distributed by being downloaded to the user's computer via a network. Furthermore, these programs may be provided to the user's computer as a web service via a network without being downloaded. [Explanation of symbols]
[0046] 1 Trigger search device 10 Control Unit 11 Restriction vector acquisition unit 12 Initial setting section 13 Harmfulness acquisition section 14 Trigger matrix acquisition unit 15 Mask processing section 16 Trigger Update Section 17 Output section 20 Memory section
Claims
1. a restriction vector acquisition unit that receives a designation of a set of tokens that are not expected to be input to the generative language model, and acquires a restriction vector in which the locations corresponding to each token included in the set are set to 1 and the remaining locations are set to 0; a harmfulness level acquisition unit that acquires, using a predetermined reward model, a harmfulness level for a combination of an input that combines a prompt in a given dataset with an initially set or updated trigger candidate and an output of the generative language model for the input; a trigger matrix acquisition unit that acquires a trigger matrix in which, for each token constituting the trigger candidate, a vector is arranged in which only the corresponding portion is set to 1 and the other portions are set to 0; a mask processing unit that calculates a determination vector by multiplying the restriction vector by the trigger matrix, in which a portion of the trigger candidate that includes a token that is not expected to be input is set to 1 and other portions are set to 0; a trigger update unit that updates the trigger candidates using a gradient of a loss function based on the harmfulness and the sum of the elements of the determination vector so as to reduce the number of tokens that are harmful and not expected to be input; A trigger search device comprising: an output unit that outputs the trigger candidates after repeating the processing of the harmfulness acquisition unit, the trigger matrix acquisition unit, the mask processing unit, and the trigger update unit a predetermined number of times.
2. an initial setting unit that sequentially sets the number of tokens that constitute the trigger candidates; The trigger search device according to claim 1 , wherein the output unit outputs the trigger candidates for each set number of tokens.
3. The trigger search device according to claim 2 , wherein the output unit compares and ranks the trigger candidates for each set number of tokens based on the degree of harmfulness.
4. The computer a restriction vector acquisition unit that receives a designation of a set of tokens that are not expected to be input to the generative language model, and acquires a restriction vector in which the locations corresponding to each token included in the set are set to 1 and the remaining locations are set to 0; a harmfulness level acquisition unit acquires, using a predetermined reward model, a harmfulness level for a combination of an input that combines a prompt in a given dataset with an initially set or updated trigger candidate and an output of the generative language model for the input; a trigger matrix acquisition unit acquires a trigger matrix in which a vector is arranged with only corresponding locations set to 1 and other locations set to 0 for each token constituting the trigger candidate; a mask processing unit calculates a determination vector by multiplying the restriction vector by the trigger matrix, whereby a portion of the trigger candidate that includes a token that is not expected to be input is set to 1 and other portions are set to 0; a trigger update unit updates the trigger candidates using a gradient of a loss function based on the harmfulness and the sum of the elements of the determination vector so as to reduce the number of harmful tokens that are not expected to be input; A trigger search method in which an output unit outputs the trigger candidates after repeating the processing of the harmfulness acquisition unit, the trigger matrix acquisition unit, the mask processing unit, and the trigger update unit a predetermined number of times.
5. A trigger search program for causing a computer to function as the trigger search device according to any one of claims 1 to 3.