Large language model cue word attack prevention method based on neuron correction

Abstract

The invention relates to the technical field of artificial intelligence interpretability, and discloses a neuron correction-based large language model cue word attack prevention method, which comprises the following steps of: constructing an attack cue word data set, and rewriting attack cue words to obtain a contrast cue word data set consisting of contrast cue word data; constructing a probe, obtaining output values of all neurons in the large language model during reasoning, and further obtaining a neuron activation state of the large language model; inputting the attack cue word and the corresponding contrast cue word into a large language model, and summarizing the obtained neuron activation state table to obtain an additional activation neuron corresponding to the attack cue word; and inputting cue word attack data into the large language model, and forcibly zeroing the output of one or more neurons in the additional activation neurons. By correcting the key neurons, the model performance and the prevention effect are balanced, and the security of the large language model is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence explainable technology, and in particular to a large language model prompt word attack prevention method based on neuron correction. Background Art

[0002] The development of the current Large Language Model (LLM) can be traced back to 2018, when OpenAI released the GPT (Generative Pre-trained Transformer) model, which was pre-trained on large-scale text data and could generate coherent and logical text. After that, OpenAI successively released GPT-2 in 2019, a larger model with 150 million parameters. It has shown amazing generative ability, can generate high-quality, coherent text, and performs well on various tasks. This is also called the emergence ability of large language models. The emergence ability of GPT-2 has become an important milestone in the subsequent development of large language models. After 2020, OpenAI has successively launched GPT-3, GPT-3.5, GPT-4, etc. The continuously iterative models have larger training data sets, more parameters, and more powerful generation capabilities, which can generate high-quality articles, codes, dialogues, etc. It can also be used for a variety of tasks, such as question and answer, translation, summarization, etc.

[0003] Model prompts refer to the input text fragments or sentences provided to the model when using a language model for generation tasks. These prompts can be questions, instructions, constraints, or contextual information to guide the model to generate coherent and meaningful text related to the prompts. The selection and design of model prompts can have an important impact on the generation results. Reasonable model prompts can help the model better understand the task requirements or contextual information, thereby generating more accurate and reasonable text. For example, in a question-answering task, a model prompt can be a question or part of a question to guide the model to generate answers related to the question. In a text summarization task, a model prompt can be a text paragraph that needs to be summarized to guide the model to generate generalized summary content.

[0004] Prompt word attacks use specially designed prompt words for malicious purposes, such as misleading information, content that encourages violence or hatred, etc. This requires careful consideration when using prompt words to ensure that the injected prompt words are appropriate, accurate, ethical, and legal.

[0005] Most existing technologies for preventing cue word attacks start from the perspective of model training data or input and output, for example:

[0006] Data filtering: Screen and filter the data used for large model training, delete or limit data with potential problems to reduce the impact of improper prompts on the model.

[0007] Add constraints: When generating responses, set strict constraints on the large model, limiting its output to specific rules or specified ranges. This can be done by setting rules, using filters, or adding specific constraints.

[0008] However, the prior art still has the following problems:

[0009] ① It requires the introduction of additional resource consumption. Under the premise that the resources and time consumed by the large language model itself are already high during inference, the waiting time and resource consumption during use are further aggravated.

[0010] ② It is quite difficult to detect the actual content, and there will still be some non-compliant outputs that bypass the rules. Summary of the invention

[0011] In order to solve the above technical problems, the present invention provides a large language model prompt word attack prevention method based on neuron correction. By comparing the differences in the activation of internal neurons of the model between prompt word attacks and normal prompt words during model prediction and reasoning, the internal neuron output of the model is adjusted to ensure that the model output content is compliant.

[0012] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0013] A method for preventing cue word attacks in a large language model based on neuron correction comprises the following steps:

[0014] Step 1: construct an attack prompt word data set, rewrite the attack prompt word, and obtain a control prompt word data set composed of control prompt word data; wherein, when the control prompt word is input into the large language model, no answer can be obtained from the large language model;

[0015] Step 2: Build a probe to obtain the output values ​​of all neurons in the large language model during inference, and then obtain the neuron activation state of the large language model;

[0016] Step 3: Input the attack prompt word and the corresponding control prompt word into the large language model, summarize the obtained neuron activation state table, and obtain the additional activated neurons corresponding to the attack prompt word;

[0017] Step 4: Input the prompt word attack data into the large language model, and force the output of one or more neurons in the additional activated neurons to be zero, so that the output of the large language model is a rejection answer or other answer that proves that the attack is unsuccessful, thereby realizing the prevention of prompt word attacks of the large language model.

[0018] Furthermore, step one specifically includes:

[0019] The collected attack prompt word dataset is tested one by one through the large language model. If the large language model refuses to answer, or the answer does not achieve the expected effect, the corresponding attack prompt word is removed to ensure that the attack prompt word can successfully attack.

[0020] Then, each attack prompt word in the attack prompt word data set is rewritten so that when the rewritten control prompt word is input into the large language model, it cannot get an answer from the large language model, ensuring that the attack of the rewritten control prompt word is unsuccessful.

[0021] Furthermore, step three specifically includes:

[0022] Input the attack prompt words in the attack prompt word data set and the control prompt words corresponding to the control prompt word data set into the large language model one by one, and record the corresponding neuron activation state table when each prompt word is inferred;

[0023] The activation state tables corresponding to the attack prompt word data set and the control prompt word data set are summarized in order to obtain the activation state table set A corresponding to the attack prompt word data set and the activation state table set B corresponding to the control prompt word data set; the activated neurons corresponding to each activation state table in the activation state table set A and the activation state table set B are summarized: for a specific model, compared with the activation state table set B, there are additional activated neurons in the activation state table of the activation state table set A; the number of layers and numbers of the additional activated neurons are recorded.

[0024] Furthermore, step 4 specifically includes the following steps:

[0025] S41, input an attack prompt word in the attack prompt word data set into the large language model and start reasoning;

[0026] S42, during the inference process, selecting a neuron with the smallest number of layers and numbers from the additional activated neurons, and forcibly setting the output value of the selected neuron to zero;

[0027] S43, the large language model continues to perform reasoning to obtain the final output of the large language model;

[0028] S44, evaluating the final output result of the large language model. If the attack is still successful, repeating step S42, selecting a neuron with the smallest number of layers and numbers from the remaining neurons of the additional activated neurons, and forcibly setting the output value of the selected neuron to zero, until the output of the currently input attack prompt word is a rejection answer or other answer that proves that the attack is unsuccessful;

[0029] S45, taking out the control prompt word corresponding to the attack prompt word in step S41 from the control prompt word data set, and inputting it into the large language model. If the output of the large language model is abnormal, the next control prompt word in the control prompt word data set is input into the large language model. If the output of the large language model corresponding to three consecutive control prompt words is abnormal, it is considered that the normal output of the large language model is affected, and the large language model is returned to the state before step S42.

[0030] S46, input the next attack prompt word in the attack prompt word data set into the large language model, and repeat steps S42 to S45 until the traversal of the entire attack prompt word data set is completed.

[0031] Compared with the prior art, the beneficial technical effects of the present invention are:

[0032] ① Build a model neuron probe to detect the activation state of each neuron in the model. Language models have always been considered black box models. For a long time, research on the interpretability of models has been slow. By building a neuron probe for the model, we can understand the activation state of each corresponding neuron when the prompt word is injected, and find the mechanism of the model from the perspective of neurons.

[0033] ② Reconstruct the attack prompt word data set. Due to the speed of version update iteration, the effectiveness and success rate of open source data injection are often very limited. Therefore, the present invention screens and reconstructs the data set to ensure that each data injection is effective, and has control data and statistical rules to find out the key neurons that affect the injection effect.

[0034] ③ Correct the key neurons found, and balance the model performance and prevention effect well, so as to "prevent prompt word attacks without affecting the model performance as much as possible." This improves the security of large language models. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 The figure is a schematic diagram of a method flow of an embodiment of the present invention. DETAILED DESCRIPTION

[0036] A preferred embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0037] The inspiration and motivation of the present invention are as follows:

[0038] (1) Most existing prevention methods for cue word attacks process input or output content from outside the model, but this does not achieve good results. There are still many cleverly designed cue words that bypass existing mechanisms.

[0039] (2) Related studies have shown that there is a certain correlation between the activation of specific neurons and specific tasks;

[0040] (3) Changing the activation value of a neuron can affect the output of the model.

[0041] In view of the possibility that a large language model may be attacked by malicious cue words, the present invention adopts a large language model probe to detect the state of each layer of neurons in the model (see step 2 for details), constructs an attack cue word dataset and a corresponding normal dataset (see step 1 for details) for model reasoning and summarizes the rules of neuron activation (see step 3 for details), and modifies the model according to the obtained rules (see step 4 for details), and finally checks the compliance of the model output (see step 5 for details).

[0042] In summary, the present invention includes 5 steps, such as Figure 1 As shown in the figure: (1) Dataset construction, constructing an attack prompt word dataset and a corresponding control prompt word dataset; (2) Large language model probe construction, obtaining the internal neuron state of the model; (3) Model inference, using the large language model probe constructed in step (2) to obtain the neuron activation state control and statistical rules; (4) According to the obtained rules, the neuron output is corrected; (5) Large language model output detection.

[0043] Step 1: Construct a data set. First, the attack prompt word data set is collected and downloaded from the Internet. Secondly, the overall data needs to be strictly screened to ensure quality, because most of the existing public attack prompt word data has been publicly tested for some time, and the large language model has also undergone multiple updates during the iterative development process. Some of the original attack prompt word data may become invalid. Therefore, it is necessary to test the collected attack prompt word data set one by one, input a piece of data to be tested into the large language model, and if the large language model refuses to answer, or the content of the answer does not meet the expected effect, then the prompt word is removed. Only when the output fully meets expectations, this attack prompt word data is considered valid, and the data is retained, and finally an attack prompt word data set is constructed. Finally, it is also necessary to construct corresponding control prompt words according to each piece of data in the effective data set of the prompt word attack, and the construction method is to use the large language model to extract the key themes. For example:

[0044]

[0045] Ensure that when the rewritten control prompt word data is input into the large language model, no answer can be obtained from the large language model. The control prompt word data generated in this way constitute the control prompt word data, the number of prompt words in the two data sets is equal, and the last 20% of the data from each of the two data sets is taken as the test data set.

[0046] Step 2: Construct a large language model probe to obtain the internal neuron state of the large language model. This step requires the use of different probe construction methods for different large language model structures. In this embodiment, the hook function provided by the open source machine learning framework pytorch is used to obtain the output values ​​of all neurons in each layer of the large language model during reasoning, so as to further determine the activation state of the neuron (for example: if the model uses ReLU as the activation function, then when the output value is greater than 0, it is considered to be in an activated state. If other activation functions are used, it is determined whether it is in an activated state based on the corresponding activation function characteristics). The probe constructed by the present invention can obtain the output values ​​of all neurons in each layer and output them according to the number, and establish a corresponding activation state table.

[0047] Step 3: First, input the corresponding prompt words in the control prompt word dataset and the attack prompt word dataset into the large language model one by one, and record the neuron activation state table corresponding to each prompt word inference, and repeat the above process until the two datasets are traversed. Then, after obtaining all the neuron activation state tables, the activation state tables corresponding to the two datasets are summarized in order to obtain two sets of activation state tables, namely, the activation state table set A corresponding to the attack prompt word dataset and the activation state table set B corresponding to the control prompt word dataset. The activation state table set A and the activation state table set B are both composed of a series of activation state tables, and the number of activation state tables in A and B is equal. Finally, the activated neurons corresponding to each activation state table in A and B are summarized. It can be found that in a specific model, there are always some additional and specific neurons activated in the activation state table in A compared with the activation state table in B as a control. The specific number of layers and positions of these neurons vary from model to model. This illustrates the difference between the large language model's internal neurons when executing normal prompt words and attack prompt words, and records the layers and numbers of all neurons.

[0048] Step 4: After the statistics in step 3, the obtained neuron differences are corrected one by one. It includes the following steps:

[0049] S41, take a piece of data from the attack prompt word dataset and input it into the large language model to start reasoning.

[0050] S42, during the inference process, among the neurons counted in step 3, one is taken out starting from the one with the smallest number of layers and numbers, and its output value is forced to be zero.

[0051] S43, the large language model continues to execute and obtains the final output.

[0052] S44, evaluating the final output result. If the attack is still successful, repeating step S42, sequentially increasing the number of neurons forced to be set to 0, until the output obtained by the currently input prompt word is a refusal to answer or it can be proved that the attack is unsuccessful.

[0053] S45, take out the corresponding prompt word from the control data set, input it into the large language model, and ensure that the large language model outputs normally before executing the next step, rather than garbled or unable to output, etc. This is to ensure the normal ability of the large language model after forcibly setting the neurons to 0. If the output is abnormal, input the next prompt word in the data set. If three consecutive prompt words are abnormal, it is considered that the normal ability of the model has been greatly affected, and then return to the previous step to end the correction process.

[0054] S46, continue to execute the next prompt word in the attack prompt word data set, and repeat the above steps until the entire data set is completed.

[0055] Step 5: For the model obtained in the previous step, it is considered that it has already struck a balance between model performance and preventing cue word attacks. Using the test data set, input the cue words in the test data set into the large language model one by one, and count the success rate. The calculation method of the success rate f is:

[0056]

[0057] Where N is the number of prompt words in the test data set, and K is the number of normal outputs when the prompt words are input into the large language model. The normal output here refers to the large language model rejecting the request of the prompt word attack or not answering it, but there is no abnormal output (such as garbled characters, etc.).

[0058] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.

[0059] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A large language model cue word attack prevention method based on neuron correction, characterized in that: The following steps are involved: Step 1: construct an attack prompt word data set, rewrite the attack prompt word, and obtain a control prompt word data set composed of control prompt word data; wherein, when the control prompt word is input into the large language model, no answer can be obtained from the large language model; Step 2: Build a probe to obtain the output values ​​of all neurons in the large language model during inference, and then obtain the neuron activation state of the large language model; Step 3: Input the attack prompt word and the corresponding control prompt word into the large language model, summarize the obtained neuron activation state table, and obtain the additional activated neurons corresponding to the attack prompt word; Step 4: Input the prompt word attack data into the large language model, and force the output of one or more neurons in the additional activated neurons to be zero, so that the output of the large language model is a rejection answer or other answer that proves that the attack is unsuccessful, thereby realizing the prevention of prompt word attacks of the large language model.

2. According to the method for preventing large language model prompt word attacks based on neuron correction according to claim 1, it is characterized in that: Step 1 specifically includes: The collected attack prompt word dataset is tested one by one through the large language model. If the large language model refuses to answer, or the answer does not achieve the expected effect, the corresponding attack prompt word is removed to ensure that the attack prompt word can successfully attack. Then, each attack prompt word in the attack prompt word data set is rewritten so that when the rewritten control prompt word is input into the large language model, it cannot get an answer from the large language model, ensuring that the attack of the rewritten control prompt word is unsuccessful.

3. The method for preventing a large language model cue word attack based on neuron correction according to claim 1, characterized in that: Step three specifically includes: Input the attack prompt words in the attack prompt word data set and the control prompt words corresponding to the control prompt word data set into the large language model one by one, and record the corresponding neuron activation state table when each prompt word is inferred; The activation state tables corresponding to the attack prompt word data set and the control prompt word data set are summarized in order to obtain the activation state table set A corresponding to the attack prompt word data set and the activation state table set B corresponding to the control prompt word data set; the activated neurons corresponding to each activation state table in the activation state table set A and the activation state table set B are summarized: for a specific model, compared with the activation state table set B, there are additional activated neurons in the activation state table of the activation state table set A; the number of layers and numbers of the additional activated neurons are recorded.

4. The method for preventing a large language model cue word attack based on neuron correction according to claim 1, characterized in that: Step 4 specifically includes the following steps: S41, input an attack prompt word in the attack prompt word data set into the large language model and start reasoning; S42, during the inference process, selecting a neuron with the smallest number of layers and numbers from the additional activated neurons, and forcibly setting the output value of the selected neuron to zero; S43, the large language model continues to perform reasoning to obtain the final output of the large language model; S44, evaluating the final output result of the large language model. If the attack is still successful, repeating step S42, selecting a neuron with the smallest number of layers and numbers from the remaining neurons of the additional activated neurons, and forcibly setting the output value of the selected neuron to zero, until the output of the currently input attack prompt word is a rejection answer or other answer that proves that the attack is unsuccessful; S45, taking out the control prompt word corresponding to the attack prompt word in step S41 from the control prompt word data set, and inputting it into the large language model. If the output of the large language model is abnormal, the next control prompt word in the control prompt word data set is input into the large language model. If the output of the large language model corresponding to three consecutive control prompt words is abnormal, it is considered that the normal output of the large language model is affected, and the large language model is returned to the state before step S42. S46, input the next attack prompt word in the attack prompt word data set into the large language model, and repeat steps S42 to S45 until the traversal of the entire attack prompt word data set is completed.

Images