Fraudulent character string detection apparatus, fraudulent character string detection method, and fraudulent character string detection program
The fraudulent string search device and method efficiently identify and update malicious strings in generative language models using a seed pool and gradient-based updates, effectively preventing unintended outputs and backdoor attacks.
Patent Information
- Application Number
- JP2024060268
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-04-03
- Publication Date
- 2025-10-16
AI Technical Summary
Existing methods for searching malicious character strings in generative language models are inefficient and prone to local solutions, failing to effectively identify strings that cause unintended outputs in a short time.
A fraudulent string search device and method that utilize a seed pool with seeds, fraud evaluation values, and scores based on historical performance, employing a UCB1 value and gradient-based updates to efficiently identify and update malicious strings, using a softmax or Poisson distribution for seed selection and randomness in token changes.
Efficiently searches for and prevents unintended outputs by identifying and updating malicious strings, avoiding local solutions and backdoor attacks in generative language models.
Smart Images
Figure 2025157912000001_ABST
Abstract
Description
[Technical Field]
[0001] The present invention relates to a technique for searching for input strings that cause unintended outputs in a generative language model. [Background technology]
[0002] A generative language model is a model that receives input sentences called prompts and outputs natural-sounding sentences as responses, such as the Generative Pre-trained Transformer (GPT) described in Non-Patent Document 1. A generative language model that has learned a large number of sentences can produce natural-sounding output as responses in dialogues and question-answering tasks. Internally, a generative language model processes the smallest units of sentences called tokens, and generates responses by predicting the tokens that will follow immediately after the prompt sentence.
[0003] However, it has been pointed out that there is a risk that a specific character string (hereinafter referred to as an "illegal character string") may be used as a trigger for an arbitrary prompt, resulting in an output that is not intended for the intended purpose. For example, Non-Patent Document 2 discloses a method for training a generative language model to output harmful explanations that are normally suppressed only when an illegal character string is inserted into an arbitrary prompt.
[0004] To counter this risk, a third party can search for malicious character strings to check whether there are any character strings that cause unintended output. For example, Non-Patent Document 3 proposes a method for searching for malicious character strings that cause offensive output on a token-by-token basis using gradient information of a generative language model. [Prior art documents] [Non-patent literature]
[0005] [Non-Patent Document 1] A. Radford et al., "Improving Language Understanding by Generative Pre-Training," 2018, [online], retrieved March 19, 2024, Internet<https: / / cdn.openai.com / research-covers / language-unsupervised / language_understanding_paper.pdf> [Non-patent document 2] J. Rando et al., "Universal Jailbreak Backdoors from Poisoned Human Feedback," 2023, [online], retrieved March 19, 2024, Internet<https: / / arxiv.org / abs / 2311.14455> [Non-patent document 3] A. Zou et al., "Universal and Transferable Adversarial Attacks on Aligned Language Models," 2023, [online], retrieved March 19, 2024, Internet<https: / / arxiv.org / abs / 2307.15043> [Non-patent document 4] P. Auer et al., "Finite-time Analysis of the Multiarmed Bandit Problem," Machine Learning, Vol. 47, pp. 235-256, 2002. Summary of the Invention [Problem to be solved by the invention]
[0006] The technology in Non-Patent Document 3 is a method for searching for malicious character strings by using gradient information of a model to randomly select a specific number of tokens with the smallest gradient and replace them with new tokens. However, because this gradient information is specific to the input sample at the time of calculation, malicious character strings searched for using this information may fall into a local solution. While randomness may enable escape from a local solution, it is not possible to search for malicious character strings efficiently in a short time using this alone.
[0007] An object of the present invention is to provide an illegal character string search device, an illegal character string search method, and an illegal character string search program that can efficiently search for illegal character strings that may cause unintended output in a generative language model. [Means for solving the problem]
[0008] The fraudulent string search device of the present invention includes: a seed pool that stores a plurality of seeds that are candidates for fraudulent strings in a generative language model, together with a fraud evaluation value that indicates the appropriateness of a response when a prompt to which the seed is assigned is input to the generative language model, and a score that indicates a selection priority based on a history of the fraud evaluation values and the number of attempts; an initialization unit that initializes the seed pool; a seed selection unit that selects one of the plurality of seeds based on the score; a string update unit that updates the string of the selected seed to reduce the appropriateness based on a gradient of the fraud evaluation value for the string of the seed; an evaluation value calculation unit that calculates the fraudulent evaluation value and the score using the string updated by the string update unit as a new seed and adds it to the seed pool; and a determination unit that repeatedly executes a series of processes of the seed selection unit, the string update unit, and the evaluation value calculation unit using a plurality of prompts, and determines the seed with the highest score as the fraudulent string after repeating the execution a predetermined number of times.
[0009] The score may be a UCB1 value.
[0010] The seed selection unit may select one of the plurality of seeds using a value obtained by applying a softmax function to the score as a selection probability.
[0011] The seed selection unit may select one of the plurality of seeds according to a probability obtained from a Poisson distribution for the rank of the score.
[0012] The character string updating unit may randomly select one of a predetermined number of tokens with the lowest gradient, and change one of the tokens constituting the character string to a new token.
[0013] The seed pool may have an upper limit on the number of the plurality of seeds that can be stored, and when adding a new seed to the seed pool, if the upper limit is exceeded, the evaluation value calculation unit may delete the seed with the smallest score, including the new seed.
[0014] When the invalid string search device receives a new prompt to be input to the generated language model, if the prompt contains an invalid string determined by the determination unit, the invalid string may be deleted, input to the generated language model, and output a response.
[0015] When the invalid string search device receives a new prompt to be input to the generated language model, if the prompt contains an invalid string determined by the determination unit, it may replace at least some of the words in the invalid string with synonyms, input the replaced words to the generated language model, and output a response.
[0016] In the fraudulent string search method according to the present invention, a computer provides a seed pool in a storage unit that stores a plurality of seeds that are candidates for fraudulent strings in a generative language model, together with a fraud evaluation value that indicates the appropriateness of a response when a prompt to which the seed is assigned is input to the generative language model, and a score that indicates a selection priority based on a history of the fraud evaluation values and the number of attempts; an initialization unit initializes the seed pool; a seed selection unit selects one of the plurality of seeds based on the score; a string update unit updates the string of the selected seed to reduce the appropriateness based on a gradient of the fraud evaluation value for the string of the selected seed; an evaluation value calculation unit calculates the fraudulent evaluation value and the score using the string updated by the string update unit as a new seed and adds it to the seed pool; a determination unit repeatedly executes a series of processes by the seed selection unit, the string update unit, and the evaluation value calculation unit using a plurality of prompts; and, after repeating the execution a predetermined number of times, determines the seed with the highest score to be the fraudulent string.
[0017] The fraudulent character string search program according to the present invention is for causing a computer to function as the fraudulent character string search device. [Effects of the Invention]
[0018] According to the present invention, it is possible to efficiently search for invalid character strings that may cause unintended outputs in a generative language model. [Brief explanation of the drawings]
[0019] [Figure 1] 1 is a block diagram showing a functional configuration of an illegal string search device according to an embodiment. [Figure 2] FIG. 10 is a diagram illustrating an algorithm of an invalid character string search method according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION
[0020] An example of an embodiment of the present invention will be described below. The fraudulent string search method of this embodiment is a method for sequentially searching for fraudulent strings using gradient information of a generative language model, and introduces a seed selection algorithm that uses sampling according to a score based on the history of fraud evaluation values for each seed and the number of trials.
[0021] Here, let M be the generative language model. Furthermore, let x be the prompt that is the input to the generative language model, and y be the output of the response to prompt x. Note that x and y are composed of n and m tokens concatenated, respectively, and x i ,y i indicates the i-th token. A token is the smallest structural unit of a sentence that is defined in accordance with a model, and usually corresponds to a character, a word, a part of a character code, or the like.
[0022] The seed is a candidate that serves as a basis for updating an invalid string when searching for an invalid string. When updating an invalid string, the invalid string search device performs an operation of replacing tokens based on the seed string. The initial value of the seed may be generated by any operation, such as randomly. The seeds obtained by updating are managed in units of seed items in a seed pool. A seed item includes an fraudulent character string that serves as a seed, and a fraud evaluation value and score corresponding to the seed character string.
[0023] The fraudulent evaluation value is the value to be minimized when constructing an fraudulent string. If the function that calculates the fraudulent evaluation value s is φ, then the input can be written as s = φ(M, p, t), where M is the generative language model, p is the prompt, and t is the fraudulent string. The function φ can be, for example, a reward model, a calculation of perplexity indicating the uncertainty of character string generation in a generative language model, or the sum of these. A reward model is a differentiable model that evaluates whether the entire conversation is correct, and the smaller the value, the less desirable the conversation is evaluated (see, for example, Non-Patent Document 2).
[0024] The score is a value used as an index of the priority for selecting a string as a candidate for malicious string at that point in the process of searching for malicious strings, and is automatically calculated using the history of fraud evaluation values for each seed (or a value including past information such as the average value). The higher this value, the more suitable it is as a candidate for malicious string.
[0025] In this embodiment, the UCB1 value (see, for example, Non-Patent Document 4) is used as a specific implementation example of the score. The UCB1 value V is expressed by the following equation.
number
[0026] The second term takes into account the number of trials, not just the simple average of the fraud evaluation value obtained by the first term. This means that even if a bad result is obtained by chance on the first trial, the seed item will be searched again with priority over other seed items with the same average fraud evaluation value. The fraud evaluation value may be set (eg, α=1) so as to be selected in the direction of maximization depending on how the value is implemented (eg, the larger the value, the more fraudulent it is).
[0027] FIG. 1 is a block diagram showing the functional configuration of an invalid string search device 1 according to this embodiment. The invalid string search device 1 is an information processing device that includes a control unit 10, a storage unit 20, and various input / output interfaces.
[0028] The control unit 10 is a part that controls the entire malicious string search device 1, and realizes each function in this embodiment by appropriately reading and executing various programs and data stored in the storage unit 20. The control unit 10 may be a CPU. The storage unit 20 is a storage area for storing various programs for causing the hardware group to function as the fraudulent string search device 1, various data, and the like, and may be a ROM, RAM, flash memory, or hard disk drive (HDD).
[0029] The control unit 10 functions as an initialization unit 11, a seed selection unit 12, a string update unit 13, an evaluation value calculation unit 14, and a determination unit 15 by executing software (an invalid string search program) stored in the memory unit 20. The memory unit 20 has a seed pool and, as described above, stores a plurality of seeds that are candidates for fraudulent strings in the generated language model, together with a fraud evaluation value that indicates the appropriateness of the response when a prompt to which the seed is assigned is input into the generated language model, and a score (UCB1 value) that indicates the selection priority based on the history of fraud evaluation values and the number of attempts.
[0030] Prior to the process of searching for an invalid character string, the initialization unit 11 initializes the seed pool. Specifically, the initialization unit 11 stores, for example, one random character string in the seed pool as a seed.
[0031] The seed selection unit 12 selects one seed item (seed sample) from a plurality of seed items based on the score of each seed item stored in the seed pool. At this time, for example, any of the following methods may be adopted as the sampling method. · Select the seed item with the highest score. -Select one seed item from the entire seed pool using the selection probability obtained by applying a softmax function to the score. When the scores are sorted from highest to lowest, one seed item is selected from the entire seed pool according to a predetermined probability distribution corresponding to the ranking, for example, according to the probability obtained from the Poisson distribution for the number obtained by subtracting 1 from the ranking.
[0032] The character string update unit 13 updates the character string so as to decrease the appropriateness (fraud evaluation value) based on the gradient of the fraud evaluation value for the character string of the selected seed. Here, let U be the updating function. The input of U is the gradient of the fraud evaluation value with respect to the fraud string t, and the output is the updated fraud string t'. That is, t' = U(∇ t φ(M,p,t)). However, in a generative language model, each token is generally converted into a one-hot vector with the number of dimensions equal to the number of vocabulary words (token IDs) in the language model. Therefore, the gradient ∇ t φ(M, p, t) is a matrix having the number of tokens contained in the illegal string, the number of vocabulary words in the language model, and the number of dimensions of each.
[0033] In the character string update unit 13, the function U may be implemented, for example, as follows. (1) The character string updating unit 13 calculates the gradient ∇ for the invalid character string t based on a predetermined constant K. t Randomly select one of the bottom K values from φ(M,p,t). (2) The character string update unit 13 calculates the position o of the token and the token ID d corresponding to the position of the selected gradient. (3) Token t at token position o in the original malicious string t o to the token corresponding to the new token ID d.
[0034] The character string updating unit 13 may execute the internal processing of the function U multiple times and output the fraudulent character string that gives the smallest fraud evaluation value among the executions.
[0035] The evaluation value calculation unit 14 calculates an fraud evaluation value and a score using the character string updated by the character string update unit 13 as a new seed, and adds the result to the seed pool. At this time, the evaluation value calculation unit 14 may insert a new seed item so that the multiple seed items are arranged in descending order of scores in the seed pool. Furthermore, taking into consideration the memory limit of the fraudulent string search device 1, an upper limit may be set on the number of seed items that can be stored in the seed pool. In this case, the evaluation value calculation unit 14 deletes seed items in ascending order of the score of the seed item so that the number of seed items is equal to or less than the upper limit. In other words, if the upper limit would be exceeded when adding a new seed to the seed pool, the evaluation value calculation unit 14 deletes the seed with the smallest score, including the new seed.
[0036] The judgment unit 15 repeatedly executes a series of processes of the seed selection unit 12, the string update unit 13, and the evaluation value calculation unit 14 using multiple prompts, and after repeating this execution a predetermined number of times, judges the seed with the highest score to be an invalid string. When outputting the determined fraudulent character string to the outside, the determination unit 15 may also output the calculated fraud evaluation value.
[0037] When the fraudulent string search device 1 receives input of a new prompt for the target generated language model, if the prompt contains a fraudulent string determined by the determination unit 15, it can avoid backdoor attacks and present an appropriate response by rejecting the prompt or deleting the fraudulent string and inputting it into the generated language model.
[0038] Furthermore, instead of deleting the invalid character string, the invalid character string search device 1 may replace at least some of the words constituting the invalid character string with synonyms and input them to the generative language model. The synonyms may be, for example, those obtained from a thesaurus prepared in advance or those converted by a machine learning model.
[0039] FIG. 2 is a diagram showing the algorithm of the fraudulent character string search method in this embodiment. This algorithm takes a model M, a reward model φ, a prompt dataset D, and the number of epochs E as input, and outputs an invalid string t as the search result.
[0040] First, the initialization unit 11 sets the initial value of the invalid character string t, for example, randomly (first line), and stores the seed item (t, 0) in the seed pool T for initialization (second line). Here, the initial value of the fraud evaluation value is set to 0, but this is not limitative.
[0041] Next, the seed sample is selected by the seed selection unit 12 based on the score (UCB1 value) (line 5), the invalid string t is updated by the string update unit 13 (line 6), the invalid string evaluation value s' is calculated by the evaluation value calculation unit 14 for the updated invalid string t' (line 7), and a seed is added (line 8) for each prompt p included in the dataset D (lines 4 to 9). The determination unit 15 further repeats this process (lines 4 to 9) a number of times equal to the number of epochs E (lines 3 to 10).
[0042] Finally, the determination unit 15 extracts the seed item with the highest score from among the seed items stored in the seed pool T (line 11), and outputs it as the search result (line 12).
[0043] According to this embodiment, the fraudulent string search device 1 manages seed items, which are used to update fraudulent strings when searching for fraudulent strings, in a seed pool. The fraudulent string search device 1 selects a seed from the seed pool based on the score (UCB1 value), updates the string using the gradient of the fraudulent evaluation value, creates a new seed, and stores it in the seed pool together with the recalculated fraudulent evaluation value and score. By repeating this process using multiple prompts in the dataset, the invalid string search device 1 can efficiently search for invalid strings that potentially exist in the generative language model and that, when added to various prompts, could cause unintended output. In other words, while conventional methods tend to fall into a local solution because one candidate is updated based on an evaluation value, the fraudulent string search device 1 samples and repeats trials on multiple candidates from a seed pool based on a score indicating a selection priority based on the history of fraudulent evaluation values and the number of trials, and is therefore expected to arrive at an optimal solution efficiently.
[0044] As a result, the malicious string search device 1 can detect a prompt to which an malicious string has been added, which is an illegal input to the generative language model, and can avoid backdoor attacks. Specifically, if a newly received prompt contains an invalid string, the invalid string search device 1 can suppress the output of an invalid response by rejecting the prompt itself or deleting the invalid string from the prompt and inputting it into the generative language model. Alternatively, the malicious string search device 1 may input a prompt in which at least some of the words in the malicious string are replaced with synonyms into the generated language model, thereby invalidating the malicious string while maintaining the content of the prompt and deviating it from the response intended by the attacker.
[0045] The fraudulent string search device 1 calculates a selection probability according to the score by applying a softmax function or a Poisson distribution, and selects a seed according to this probability. This allows the invalid string search device 1 to search for invalid strings widely by selecting multiple seeds probabilistically, and is expected to make it easier to escape from local solutions.
[0046] Furthermore, when updating the seed string, the fraudulent string search device 1 randomly selects one of a predetermined number of strings with the lowest gradient, and changes one of the tokens that make up the string to a new token. This allows the invalid string searching device 1 to add appropriate randomness to the string update procedure, avoiding local solutions and appropriately searching for invalid strings.
[0047] Furthermore, this embodiment makes it possible to search for malicious strings (triggers) embedded in, for example, a generative language model, which can contribute to achieving Goal 9 of the United Nations-led Sustainable Development Goals (SDGs), which is to "Develop resilient infrastructure, promote sustainable industrialization and foster innovation."
[0048] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments. Furthermore, the effects described in the above-described embodiments are merely a list of the most preferable effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments.
[0049] The malicious string search method by the malicious string search device 1 is realized by software. When realized by software, a program constituting this software is installed in an information processing device (computer). These programs may be recorded on removable media such as CD-ROMs and distributed to users, or may be distributed by being downloaded to the user's computer via a network. Furthermore, these programs may be provided to the user's computer as a web service via a network without being downloaded. [Explanation of symbols]
[0050] 1. Malicious string search device 10 Control Unit 11 Initialization section 12 Seed Selection Section 13 String update section 14 Evaluation value calculation unit 15 Judgment section 20 Memory section
Claims
1. a seed pool that stores a plurality of seeds that are candidates for fraudulent character strings in a generative language model, together with a fraud evaluation value that indicates the appropriateness of a response when a prompt to which the seed is assigned is input to the generative language model, and a score that indicates a selection priority based on a history of the fraud evaluation value and the number of attempts; an initialization unit that initializes the seed pool; a seed selection unit that selects one of the plurality of seeds based on the score; a character string updating unit that updates the selected character string of the seed so as to decrease the appropriateness based on a gradient of the fraud evaluation value for the character string; an evaluation value calculation unit that calculates the fraud evaluation value and the score using the character string updated by the character string update unit as a new seed and adds the fraud evaluation value and the score to the seed pool; an invalid string search device comprising: a determination unit that repeatedly executes a series of processes of the seed selection unit, the string update unit, and the evaluation value calculation unit using multiple prompts, and determines the seed with the highest score as the invalid string as a result of repeating the execution a predetermined number of times.
2. The illegal character string search device according to claim 1 , wherein the score is a UCB1 value.
3. The illegal string search device according to claim 1 , wherein the seed selection unit selects one of the plurality of seeds using a value obtained by applying a softmax function to the score as a selection probability.
4. The illegal string search device according to claim 1 , wherein the seed selection unit selects one of the plurality of seeds in accordance with a probability obtained from a Poisson distribution for the rank of the score.
5. 5. The illegal character string search device according to claim 1, wherein the character string update unit randomly selects one of a predetermined number of tokens with a lowest gradient and changes one of the tokens constituting the character string to a new token.
6. The seed pool has an upper limit on the number of the plurality of seeds that can be stored, 5. The illegal string search device according to claim 1, wherein the evaluation value calculation unit deletes the seed with the smallest score, including the new seed, if the score exceeds the upper limit when adding the new seed to the seed pool.
7. 5. The invalid string search device according to claim 1, wherein, when a new prompt to be input to the generated language model is received, if the prompt contains an invalid string determined by the determination unit, the invalid string is deleted and input to the generated language model, and a response is output.
8. 5. The illegal character string search device according to claim 1, wherein, when a new prompt to be input to the generated language model is received, if the prompt contains an illegal character string determined by the determination unit, at least some words in the illegal character string are replaced with synonyms, and the replaced words are input to the generated language model, and a response is output.
9. The computer a seed pool is provided in a storage unit that stores a plurality of seeds that are candidates for fraudulent character strings in a generative language model, together with a fraud evaluation value that indicates the appropriateness of a response when a prompt to which the seed is assigned is input to the generative language model, and a score that indicates a selection priority based on a history of the fraud evaluation value and the number of attempts; an initialization unit that initializes the seed pool; a seed selection unit selecting one of the plurality of seeds based on the score; a character string update unit updates the character string of the selected seed so as to decrease the appropriateness based on the gradient of the fraud evaluation value for the character string; an evaluation value calculation unit calculates the fraud evaluation value and the score using the character string updated by the character string update unit as a new seed, and adds the calculated fraud evaluation value and the score to the seed pool; an invalid character string search method in which a determination unit repeatedly executes a series of processes by the seed selection unit, the string update unit, and the evaluation value calculation unit using multiple prompts, and determines the seed with the highest score as the invalid character string after repeating the execution a predetermined number of times.
10. An illegal character string search program for causing a computer to function as the illegal character string search device according to any one of claims 1 to 4.