A gradient search attack method based on prompt template
By constructing templates using gradient search and token sequences generated by GPT-2, the security vulnerability of Prompt-based models was addressed, resulting in a significant reduction in model prediction accuracy and demonstrating the effectiveness of the attack.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-19
- Publication Date
- 2026-04-03
AI Technical Summary
Existing Prompt-based models are vulnerable to security attacks in the field of natural language processing, especially generative adversarial examples and backdoor attacks, which affect the accuracy of model predictions and pose potential risks, particularly in security-sensitive tasks.
A gradient search-based attack method is adopted, which automatically finds the tag mapping words, constructs a candidate word set, and selects the token sequence that will reduce the model accuracy to construct a template. This includes automatic tag selection, gradient search, and template generation, and uses the GPT-2 model to generate a highly concealed token sequence.
It effectively reduced the prediction accuracy of the pre-trained model. Experiments show that the token sequence generated by the GPT-2 method has the lowest perplexity, proving the effectiveness of its attack and successfully reducing the model accuracy by 56.77%.
Smart Images
Figure CN115358224B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of information security technology, specifically a gradient search attack method based on prompt templates. Background Technology
[0002] Prompt-based models consist of three steps: First, we need a pre-trained model whose pre-training task includes Masked Language Models (MLMs), such as Bert and Roberta. Second, to adapt downstream tasks to resemble the pre-training task, we need to construct a cloze test template. For example, in a text sentiment classification task, for the sentence "This is the best movie I've ever watched.", we can construct the template "The movie is [MASK]", and then use a Prompt Model (PLM) to fill in the blanks, predicting which sentiment word (e.g., "great", "wonderful") should be filled in. Finally, the PLM-predicted answers are converted into sentiment classification labels; words like "great" correspond to "positive," while words like "terrible" correspond to "negative." We call this correspondence a verbbalizer.
[0003] Natural Language Processing (NLP) encompasses a wide range of tasks, from high-accuracy tasks like text classification to those involving information security, such as recommender systems. In recent years, deep learning-based text analysis and understanding have become the core technology behind various NLP applications. However, despite their popularity and excellent performance, research indicates that deep neural network-based NLP models are vulnerable to malicious attacks. Given their increasingly widespread application in many real-world security-sensitive tasks such as sentiment analysis and harmful content detection, this vulnerability has raised significant concerns and high levels of attention regarding the security of NLP models. Therefore, the security of NLP models has gradually become a new research hotspot, with more and more researchers focusing on the attack and defense strategies for NLP models.
[0004] While the Prompt model has demonstrated outstanding performance in various tasks within natural language processing, including text understanding and text generation, and has broad application prospects, its security has not been fully and thoroughly evaluated. Deploying and using such models in real-world systems may pose significant security risks, and using these models without sufficient awareness of potential security risks could lead to serious consequences. Therefore, in-depth research into the security issues of the Prompt model is essential. In recent years, adversarial attacks and backdoor attacks have become increasingly serious, becoming the primary threat in the field of artificial intelligence security, attracting widespread attention and in-depth research from academia and industry. We believe that common generative adversarial example attack methods and backdoor attack methods are also applicable to prompt-based models. However, we believe that the unique component of this method—the template—is crucial. Many papers have verified that template selection has a significant impact on model prediction results. Therefore, theoretically, if we construct a sufficiently unfriendly model, we can significantly reduce the accuracy of model predictions. Inspired by existing methods for automatically generating prompts, we propose our own attack method. Summary of the Invention
[0005] To address the technical problems mentioned in the background section, this invention provides an attack method targeting the prompt template portion. The main idea is to select tokens that can lower the final prediction result to construct the template; how to determine the final token is the key to the method.
[0006] To solve its technical problem, the present invention adopts the following technical solution:
[0007] A gradient search attack method based on prompt templates includes the following steps:
[0008] (1) Automatically find tag mapping words: Generate a set of words corresponding to tags through automatic tag selection method;
[0009] (2) Constructing a candidate word set: Using a gradient-based search method, the 100 words most likely to cause the model to make a prediction error are found in the word set corresponding to the labels generated in step (1) to form a candidate word set;
[0010] (3) Template construction: Select one or more words from the candidate word set, construct a token sequence, and select the sequence that causes the model accuracy to drop the most as the final template.
[0011] Furthermore, step (1) of automatically finding tag mapping words includes the following steps:
[0012] (11) Train a logistic classifier that uses a context containing the [MASK] tag as input to predict the class label, and write the output of this classifier as:
[0013]
[0014]
[0015] Among them, h (i) Indicates that the transformer is paired with... The encoded results, y and β y These are the learning weights and biases, where i represents the index of [MASK]. Indicates the context containing the [MASK] tag;
[0016] (12) h (i) Replace with the embedding of the output word in PLM. out To obtain the score s(y,w)=p(y|w) out Let s(y,w) represent the relationship between token w and label y; select the k highest-scoring words from vocabulary V to construct a set of words corresponding to the labels:
[0017]
[0018] Step (2) involves constructing the candidate word set, which includes the following steps:
[0019] (21) The predicted probability of the true label is represented as p(y|X). prompt ):
[0020]
[0021] (22) Replace the j-th token in the trigger token sequence with each token in the vocabulary V in turn. Let w represent the token. The first-order approximation of the loss change in this replacement process is:
[0022]
[0023] Where w in The input embedding of token w is used; the j-th token in the trigger token sequence is replaced by each token in the vocabulary V, the loss of token w with respect to the real label y is calculated, and then the gradient of token w is obtained through backpropagation.
[0024] (23) In order to achieve the effect of ultimately causing the prediction result to be wrong, the k words that minimize the approximation value are selected as the candidate word set V of the trigger token. cand :
[0025]
[0026] Furthermore, the template construction step (3) includes the following steps:
[0027] (31) Use beam search, random replacement or GPT-2 method to generate multiple token sequences to form a template, evaluate the accuracy of all templates on the training set, and select the template with the lowest accuracy as the final template.
[0028] (32) Evaluate the accuracy of the final template on the test set.
[0029] Furthermore, the beam search in step (31) includes the following steps:
[0030] All trigger tokens are initialized to [MASK], and beam search is used to select and replace [MASK] tokens from left to right; the beam width is set to k.
[0031] Within the first time step, all candidate words replace the [MASK] at the first position in turn, and are evaluated on the training set. The k words that make the attack effect on the training set the best are selected as the candidate words at the first position of the token sequence, and are used as the k candidate sequences under this time step.
[0032] In the i-th time step, based on the token candidate sequence of the previous step, all candidate words replace the [MASK] at the i-th position of each candidate sequence in turn, select the k sequences with the best attack effect as the candidate sequences for this time step, and finally select the optimal sequence from the k candidates.
[0033] Furthermore, the random replacement described in step (31) includes the following steps:
[0034] In each iteration, a position in the token sequence is randomly selected, and the token at that position is replaced with each of the candidate words in turn;
[0035] Then, evaluate the performance of these token sequences, select the template with the lowest classification accuracy on the training set in the current iteration, and calculate its result on the test set as the best result of this iteration.
[0036] After all iterations are completed, the template that produces the worst prediction results on the test set is selected as the final template.
[0037] Furthermore, the GPT-2 method described in step (31) includes the following steps:
[0038] First, a GPT-2 model is trained using all the text in the training dataset. Then, each token in the candidate word set is used as the first token. The trained GPT-2 model is used to predict the next word based on the first token until the number of tokens equals the number specified by the template. Then, the template with the lowest prediction accuracy is selected, and its perplexity value is output. The lower the perplexity value, the stronger the readability and the stronger the concealment.
[0039] Compared with the prior art, the present invention, employing the above technical solution, has the following beneficial effects:
[0040] This paper presents a method for attacking the template portion of a model based on gradient search, which fully leverages relevant technological advancements in the field of natural language processing. Experiments show that three malicious template construction methods can effectively reduce the prediction accuracy of the pre-trained model on the dataset, meaning that the attack objective can be achieved. Secondly, we evaluate the perplexity value of the generated templates. The experimental results show that the token sequence generated by the GPT-2 method has the lowest perplexity, demonstrating the effectiveness of the GPT-2 method. Attached Figure Description
[0041] Figure 1 This is a schematic diagram of the overall process of the proposed algorithm framework.
[0042] Figure 2 This is a schematic diagram of the beam search method process. Detailed Implementation
[0043] The invention will now be described in further detail with reference to the accompanying drawings.
[0044] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.
[0045] like Figure 1 As shown, this invention designs a method for attacking prompt templates. This invention searches for general adversarial triggers: token sequences without real meaning, adds them to the designed template, and when connected to any input from the dataset, can trigger this token sequence, causing the model's final label prediction to be incorrect. The search iteratively updates the tokens in the trigger sequence to increase the probability of target prediction for a batch of samples.
[0046] This invention primarily targets sentiment analysis tasks. Its steps include: 1. Automatically finding tag-mapped vocabulary: generating a set of tags corresponding to words using an automatic tag selection method; 2. Selecting a set of candidate tokens to form a template based on gradient search (here, our selected candidate set includes 100 candidate tokens); 3. Selecting a specified number of tokens from the candidate set to form a template (here, we specify that the template includes 5 trigger tokens); 4. Evaluating each template and selecting the template that results in the lowest model prediction accuracy as the final template (here, we evaluate on the training set, select the final template, and evaluate its accuracy on the test set; the difference between this value and the accuracy of the control scheme can be calculated as the attack success rate).
[0047] Step 1: Automatically find tag mapping words: Generate a set of words corresponding to tags using an automatic tag selection method;
[0048] (11) Train a logistic classifier that uses a context containing the [MASK] tag as input to predict the class label, and write the output of this classifier as:
[0049]
[0050] p(y|h (i) )∝exp(h (i) ·y+β y )
[0051] Among them, h (i) Indicates that the transformer is paired with... The encoded results, y and β y These are the learning weights and biases, where i represents the index of [MASK]. Indicates the context containing the [MASK] tag;
[0052] (12) h (i) Replace with the embedding of the output word in PLM. out To obtain the score s(y,w)=p(y|w) out Let s(y,w) represent the relationship between token w and label y; select the k highest-scoring words from vocabulary V to construct a set of words corresponding to the labels:
[0053]
[0054] Step 2: Select the set of candidate tokens to form the template based on gradient search:
[0055] Initialize all tokens to [MASK]. Note that the trigger sequence cannot be separated; it needs to be concatenated. This token sequence is then appended to the end of the original input sentence. Since the tokens are discrete, we cannot directly apply adversarial attack methods from computer vision to determine how to replace the current token. Instead, we use a gradient search approach.
[0056] We select a candidate word set by evaluating the change each word brings to the model's prediction. We use the product of the gradient and the word's input embedding as a first-order approximation of this change. The larger the first-order approximation, the more likely the model's prediction will be the true label if the current position is replaced with the selected word. As an attack method, we select words that make the approximation smaller as the candidate word set to form the sequence.
[0057] The predicted probability of the true label is represented as p(y|X). prompt ):
[0058]
[0059] The first-order approximation of the loss change in this replacement process is as follows: Each token in the vocabulary V is used to sequentially replace the j-th token in the trigger token sequence, where w represents the token.
[0060]
[0061] Where w in The input embedding of token w is used; the j-th token in the trigger token sequence is replaced by each token in the vocabulary V, the loss of token w with respect to the real label y is calculated, and then the gradient of token w is obtained through backpropagation.
[0062] To achieve the effect of ultimately causing the prediction result to be incorrect, the k words that minimize the approximation value are selected as the candidate word set V for the trigger token. cand :
[0063]
[0064] Step 3: Select a specified number of tokens from the candidate word set to construct a template:
[0065] We choose the following trigger sequence lengths: longer ones are more effective, while shorter ones are more concealed. In the specific implementation of this invention, the selected sequence length is 5.
[0066] When choosing which word in the candidate word set should replace the token at each position, we used random replacement, beam search, or GPT-2 generation methods.
[0067] The random replacement includes the following steps:
[0068] In each iteration, a position in the token sequence is randomly selected, and the token at that position is replaced with each of the candidate words in turn;
[0069] Then, evaluate the performance of these token sequences, select the template with the lowest classification accuracy on the training set in the current iteration, and calculate its result on the test set as the best result of this iteration.
[0070] After all iterations are completed, the template that produces the worst prediction results on the test set is selected as the final template.
[0071] The beam search includes the following steps:
[0072] All trigger tokens are initialized to [MASK], and beam search is used to select and replace [MASK] tokens from left to right; the beam width is set to k.
[0073] Within the first time step, all candidate words replace the [MASK] at the first position in turn, and are evaluated on the training set. The k words that make the attack effect on the training set the best are selected as the candidate words at the first position of the token sequence, forming the k candidate sequences under this time step.
[0074] In the i-th time step, based on the token candidate sequence of the previous step, all candidate words replace the [MASK] at the i-th position of each candidate sequence in turn, select the k sequences with the best attack effect as the candidate sequences for this time step, and finally select the optimal sequence from the k candidates.
[0075] The GPT-2 method includes the following steps:
[0076] First, a GPT-2 model is trained using all the text in the training dataset. Then, each token in the candidate word set is used as the first token. The trained GPT-2 model is used to predict the next word based on the first token until the number of tokens equals the number specified by the template. Then, the template with the lowest prediction accuracy is selected, and its perplexity value is output. The lower the perplexity value, the stronger the readability and the stronger the concealment.
[0077] Step 4: Evaluate each template and select the one that results in the lowest prediction accuracy for the model as the final template:
[0078] Regardless of the method used, we will generate multiple different token sequences to form a template. Finally, we will select the template that results in the worst classification of the current dataset as the final template and calculate the prediction accuracy of this template on the test. We will then calculate the difference between this accuracy and the accuracy of the control scheme as the attack success rate.
[0079] After completing the above steps, a malicious template can be generated, thereby reducing the model's accuracy.
[0080] Ultimately, the proposed solution achieves a 56.77% accuracy reduction on the SST-2 dataset with the Roberta-large model as the pre-trained model, meaning the attack success rate is 56.77%. To evaluate the effectiveness of the GPT-2 method, we assessed the perplexity of token sequences generated by the three methods under the same conditions, as shown in Table 1. The results demonstrate that the token sequences generated by the GPT-2 method can reduce model accuracy, thus enabling successful attacks. Furthermore, the token sequences generated by the GPT-2 method exhibit the lowest perplexity, proving the effectiveness of the GPT-2 method.
[0081] Table 1 shows the perplexity, prediction accuracy, and generated token sequences of the GPT-2 method.
[0082]
Claims
1. A gradient search attack method based on prompt templates, characterized in that, Includes the following steps: (1) Automatically find tag mapping words: Generate a set of words corresponding to tags through an automatic tag selection method; (2) Constructing a candidate word set: Using a gradient-based search method, the 100 words most likely to cause the model to make a prediction error are found in the word set corresponding to the labels generated in step (1) to form a candidate word set; (3) Template construction: Select one or more words from the candidate word set, construct a token sequence, and select the sequence that causes the model accuracy to drop the most as the final template; Step (1) of automatically finding tag mapping words includes the following steps: (11) Train a logistic classifier that uses a context containing the [MASK] tag as input to predict the class label. Write the output of this classifier as: Among them, h (i) Indicates that the transformer is paired with... The result of encoding and These are the learning weights and biases, where i represents the index of [MASK]. Indicates the context containing the [MASK] tag; (12) h (i) Replace with the embedding of the output word of PLM out To obtain the score s(y, w) = p(y|w) out Let s(y,w) represent the relationship between token w and label y; select the k highest-scoring words from vocabulary V to construct a set of words corresponding to the labels: ； Step (2) involves constructing the candidate word set, which includes the following steps: (21) The predicted probability of the true label is represented as p(y|X). prompt ): (22) Replace the j-th token in the trigger token sequence with each token in the vocabulary V in turn. Let w represent the token. The first-order approximation of the loss change in this replacement process is: Where w in The input embedding of token w is used; the j-th token in the trigger token sequence is replaced with each token in the vocabulary V, the loss of token w with respect to the real label y is calculated, and then the gradient of token w is obtained through backpropagation. ; (23) In order to achieve the effect of ultimately causing the prediction result to be wrong, the k words that minimize the approximation value are selected as the candidate word set V of the trigger token. cand : 。 2. The gradient search attack method based on prompt template according to claim 1, characterized in that, Step (3) involves the following steps in constructing the template: (31) Use beam search, random replacement or GPT-2 method to generate multiple token sequences to form a template, evaluate the accuracy of all templates on the training set, and select the template with the lowest accuracy as the final template. (32) Evaluate the accuracy of the final template on the test set.
3. The gradient search attack method based on prompt template according to claim 2, characterized in that, The beam search in step (31) includes the following steps: All trigger tokens are initialized to [MASK], and beam search is used to select and replace [MASK] tokens from left to right; the beam width is set to k. Within the first time step, all candidate words replace the [MASK] at the first position in turn, and are evaluated on the training set. The k words that make the attack effect on the training set the best are selected as the candidate words at the first position of the token sequence, and are used as the k candidate sequences under this time step. In the i-th time step, based on the candidate sequence of the previous step, all candidate words replace the [MASK] at the i-th position of each candidate sequence in turn, select the k sequences with the best attack effect as the candidate sequences for this time step, and finally select the optimal sequence from the k candidates.
4. The gradient search attack method based on prompt template according to claim 2, characterized in that, The random replacement in step (31) includes the following steps: In each iteration, a position in the token sequence is randomly selected, and the token at that position is replaced with each of the candidate words in turn; Then, evaluate the performance of these token sequences, select the template with the lowest classification accuracy on the training set in the current iteration, and calculate its result on the test set as the best result of this iteration. After all iterations are completed, the template that produces the worst prediction results on the test set is selected as the final template.
5. The gradient search attack method based on prompt template according to claim 2, characterized in that, The GPT-2 method described in step (31) includes the following steps: First, a GPT-2 model is trained using all the text in the training dataset. Then, each token in the candidate word set is used as the first token. The trained GPT-2 model is used to predict the next word based on the first token until the number of tokens equals the number specified by the template. Then, the template with the lowest prediction accuracy is selected, and its perplexity value is output. The lower the perplexity value, the stronger the readability and the stronger the concealment.