A large language model hint word attack prevention method based on neuron correction

Abstract

The present invention relates to the field of artificial intelligence explainable technology and discloses a method for preventing cue word attacks in a large language model based on neuron correction, comprising: constructing an attack cue word dataset, rewriting the attack cue words to obtain a control cue word dataset composed of control cue word data; constructing probes to obtain the output values ​​of all neurons in the large language model during inference, thereby obtaining the neuron activation state of the large language model; inputting the attack cue words and corresponding control cue words into the large language model, summarizing the obtained neuron activation state table, and obtaining additional activated neurons corresponding to the attack cue words; inputting the cue word attack data into the large language model, and forcibly setting the output of one or more of the additional activated neurons to zero. By correcting key neurons, the model performance and prevention effect are balanced, and the security of the large language model is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence explainable technology, and in particular to a large language model prompt word attack prevention method based on neuron correction. Background Art

[0002] The development of the current Large Language Model (LLM) can be traced back to 2018, when OpenAI released the GPT (Generative Pre-trained Transformer) model. This model, pre-trained on large amounts of text data, can generate coherent and logical text. Subsequently, OpenAI released GPT-2 in 2019, an even larger model with 150 million parameters. Its remarkable generative power, capable of generating high-quality, coherent text, and excelling on a variety of tasks, is known as the emergent power of large language models. The emergent power of GPT-2 became a significant milestone in the subsequent development of large language models. Since 2020, OpenAI has successively released GPT-3, GPT-3.5, and GPT-4. These successive models have larger training datasets, more parameters, and more powerful generative capabilities, enabling them to generate high-quality articles, code, dialogues, and more. They can also be used for a variety of tasks, such as question-answering, translation, and summarization.

[0003] Model prompts refer to the input text fragments or sentences provided to the model when using a language model for generation tasks. These prompts can be questions, instructions, constraints, or contextual information, which are used to guide the model to generate coherent and meaningful text related to the prompts. The selection and design of model prompts can have a significant impact on the generation results. Reasonable model prompts can help the model better understand the task requirements or contextual information, thereby generating more accurate and reasonable text. For example, in question-answering tasks, the model prompt can be a question or part of a question to guide the model to generate answers related to the question. In text summarization tasks, the model prompt can be a text paragraph that needs to be summarized to guide the model to generate general summary content.

[0004] Prompt word attacks use specially designed prompt words for malicious purposes, such as misleading information, content that encourages violence or hatred, etc. This requires careful consideration when using prompt words to ensure that the injected prompt words are appropriate, accurate, ethical, and legal.

[0005] Most existing technologies for preventing prompt word attacks start from the perspective of model training data or input and output, for example:

[0006] Data filtering: Screen and filter the data used for large model training, delete or limit data with potential problems to reduce the impact of improper prompts on the model.

[0007] Add constraints: When generating responses, set strict constraints on the model, restricting its output to specific rules or specified ranges. This can be achieved by setting rules, using filters, or adding specific constraints.

[0008] However, the existing technology still has the following problems:

[0009] ① It requires the introduction of additional resource consumption. Under the premise that the resources and time consumed by the large language model itself are already high during inference, it further increases the waiting time and resource consumption during use.

[0010] ② It is quite difficult to detect actual content, and some non-compliant outputs will still bypass the rules. Summary of the Invention

[0011] To solve the above technical problems, the present invention provides a large language model prompt word attack prevention method based on neuron correction. By comparing the differences in the activation of internal neurons in the model between prompt word attacks and normal prompt words during model prediction and inference, the internal neuron output of the model is adjusted to ensure that the model output content is compliant.

[0012] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0013] A method for preventing cue word attacks in a large language model based on neuron correction includes the following steps:

[0014] Step 1: Construct an attack prompt word dataset, rewrite the attack prompt words, and obtain a control prompt word dataset consisting of control prompt word data; wherein, when the control prompt words are input into the large language model, no answer can be obtained from the large language model;

[0015] Step 2: Build a probe to obtain the output values ​​of all neurons in the large language model during inference, and then obtain the neuron activation state of the large language model;

[0016] Step 3: Input the attack prompt word and the corresponding control prompt word into the large language model, summarize the obtained neuron activation state table, and obtain the additional activated neurons corresponding to the attack prompt word;

[0017] Step 4: Input the prompt word attack data into the large language model, and by forcing the output of one or more neurons in the additional activated neurons to zero, the output of the large language model is a rejection answer or other answer that proves that the attack is unsuccessful, thereby realizing the prevention of prompt word attacks on the large language model.

[0018] Furthermore, step one specifically includes:

[0019] The collected attack prompt word dataset is tested one by one using the large language model. If the large language model refuses to answer, or the answer does not meet the expected effect, the corresponding attack prompt word is removed to ensure that the attack prompt word can successfully attack.

[0020] Then, each attack prompt word in the attack prompt word dataset is rewritten so that when the rewritten control prompt word is input into the large language model, it cannot get an answer from the large language model, ensuring that the attack of the rewritten control prompt word is unsuccessful.

[0021] Furthermore, step three specifically includes:

[0022] Input the attack prompt words in the attack prompt word dataset and the control prompt words corresponding to the control prompt word dataset into the large language model one by one, and record the corresponding neuron activation state table when inferring each prompt word;

[0023] The activation state tables corresponding to the attack prompt word dataset and the control prompt word dataset are summarized in order to obtain the activation state table set A corresponding to the attack prompt word dataset and the activation state table set B corresponding to the control prompt word dataset; the activated neurons corresponding to each activation state table in the activation state table set A and the activation state table set B are summarized: for a specific model, compared with the activation state table set B, there are additional activated neurons in the activation state table of the activation state table set A; the number of layers and numbers of the additional activated neurons are recorded.

[0024] Furthermore, step four specifically includes the following steps:

[0025] S41, input an attack prompt word from the attack prompt word dataset into the large language model and start reasoning;

[0026] S42, during the inference process, selecting a neuron with the smallest number of layers and the smallest number from the additional activated neurons, and forcibly setting the output value of the selected neuron to zero;

[0027] S43, the large language model continues to reason and obtains the final output of the large language model;

[0028] S44, evaluating the final output of the large language model. If the attack is still successful, repeating step S42, selecting a neuron with the smallest number of layers and numbers from the remaining neurons of the additional activated neurons, and forcibly setting the output value of the selected neuron to zero, until the output of the currently input attack prompt word is a rejection answer or other answer that proves that the attack is unsuccessful;

[0029] S45: Extract the control prompt word corresponding to the attack prompt word in step S41 from the control prompt word dataset and input it into the large language model. If the large language model outputs an abnormality, input the next control prompt word from the control prompt word dataset into the large language model. If the large language model outputs abnormalities for three consecutive control prompt words, the normal output of the large language model is considered to be affected, and the large language model is returned to the state before step S42.

[0030] S46: Input the next attack prompt word in the attack prompt word dataset into the large language model, and repeat steps S42 to S45 until the entire attack prompt word dataset is traversed.

[0031] Compared with the prior art, the beneficial technical effects of the present invention are:

[0032] ① Build a model neuron probe to detect the activation state of each neuron in the model. Language models have long been considered black-box models, and research on model interpretability has long been slow. By building a neuron probe for the model, we can understand the activation state of each corresponding neuron when the prompt word is injected, and explore the model's mechanism of action from the perspective of neurons.

[0033] ② Reconstruct the attack prompt word dataset. Due to the speed of version update iteration, the effectiveness and success rate of open source data injection are often very limited. Therefore, the present invention screens and reconstructs the dataset to ensure that each data injection is effective, and has control data and statistical rules to find the key neurons that affect the injection effect.

[0034] ③ Correcting the key neurons found strikes a good balance between model performance and prevention effectiveness, achieving the goal of "preventing cue word attacks while minimizing impact on model performance." This improves the security of large language models. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Schematic diagram of a method flow in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] A preferred embodiment of the present invention will be described in detail below with reference to the accompanying drawings.

[0037] The inspiration and motivation of the present invention are as follows:

[0038] (1) Most existing methods for preventing prompt word attacks process input or output content from outside the model, but this does not achieve good results. There are still many cleverly designed prompt words that circumvent existing mechanisms.

[0039] (2) Related studies have shown that there is a certain correlation between the activation of specific neurons and specific tasks;

[0040] (3) Changing the activation value of a neuron can affect the output of the model.

[0041] To address the possibility of large language models being attacked by malicious cue words, the present invention uses a large language model probe to detect the state of neurons in each layer of the model (see step 2 for details). By constructing an attack cue word dataset and a corresponding normal dataset (see step 1 for details) for model reasoning and comparing and summarizing the rules of neuron activation (see step 3 for details), the model is modified according to the obtained rules (see step 4 for details), and finally the compliance of the model output is checked (see step 5 for details).

[0042] In summary, the present invention includes 5 steps, such as Figure 1 As shown in the figure: (1) Dataset construction, constructing an attack prompt word dataset and a corresponding control prompt word dataset; (2) Large language model probe construction, obtaining the internal neuron state of the model; (3) Model inference, using the large language model probe constructed in step (2) to obtain the neuron activation state control and statistical rules; (4) According to the obtained rules, the neuron output is corrected; (5) Large language model output detection.

[0043] Step 1: Construct a dataset. First, the attack prompt word dataset is collected and downloaded from the internet. Second, the entire data must be rigorously screened to ensure quality. This is because much of the publicly available attack prompt word data has been available for testing for some time, and the large language model has undergone multiple updates during its iterative development. Some existing attack prompt word data may no longer be valid. Therefore, each collected attack prompt word dataset must be tested individually. A piece of test data is fed into the large language model. If the large language model refuses to answer, or if the answer does not meet the expected result, the prompt word is discarded. Only when the output fully meets the expected result is the attack prompt word data considered valid and retained, ultimately constructing an attack prompt word dataset. Finally, a corresponding comparison prompt word is constructed for each piece of data in the valid prompt word attack dataset. This is done by extracting key themes using the large language model. For example:

[0044]

[0045] The rewritten control prompt word data is ensured to not receive an answer from the large language model when it is input into the large language model. The control prompt word data generated in this way constitutes the control prompt word data set. The number of prompt words in the two datasets is equal. The bottom 20% of the data from each dataset is taken as the test dataset.

[0046] Step 2: Construct a large language model probe to obtain the internal neuron state of the large language model. This step requires different probe construction methods for different large language model structures. In this embodiment, the hook function provided by the open source machine learning framework pytorch is used to obtain the output values ​​of all neurons in each layer of the large language model during inference, so as to further determine the activation state of the neurons (for example: if the model uses ReLU as the activation function, then when the output value is greater than 0, it is considered to be in an activated state. If other activation functions are used, whether it is in an activated state is determined according to the corresponding activation function characteristics). The probe constructed by the present invention can obtain the output values ​​of all neurons in each layer and output them according to the number, and establish a corresponding activation state table.

[0047] Step 3: First, the corresponding cue words in the control cue word dataset and the attack cue word dataset are input into the large language model one by one. The corresponding neuron activation table for each cue word inference is recorded. This process is repeated until both datasets are traversed. Then, after all neuron activation tables are obtained, the activation tables corresponding to the two datasets are sequentially summarized to obtain two sets of activation tables: Set A for the attack cue word dataset and Set B for the control cue word dataset. Both Set A and Set B consist of a series of activation tables, with A and B each containing an equal number of activation tables. Finally, the activated neurons corresponding to each activation table in A and B are summarized. It can be found that in a specific model, the activation table in A always has some additional, specific neurons activated compared to the activation table in B, which serves as the control. The specific number of layers and locations of these neurons vary from model to model. This illustrates the difference in neurons within the large language model when executing normal prompt words and attack prompt words, and records the number of layers and numbers of all neurons.

[0048] Step 4: After the statistics in step 3, the obtained neuron differences are corrected one by one. This includes the following steps:

[0049] S41: Take a piece of data from the attack prompt word dataset and input it into the large language model to start reasoning.

[0050] S42, during the inference process, among the neurons counted in step 3, starting from the one with the smallest layer number and number, take out one and force its output value to be zero.

[0051] S43: The large language model continues to execute and obtains the final output.

[0052] S44, evaluate the final output result. If the attack is still successful, repeat step S42 and increase the number of neurons forced to be 0 in sequence until the output of the currently input prompt word is a refusal to answer or it can be proved that the attack is unsuccessful.

[0053] S45: Extract the corresponding prompt word from the control dataset and input it into the large language model. Ensure the large language model outputs normally, rather than garbled or unavailable output, before proceeding to the next step. This is to ensure the normal performance of the large language model after forcing the neurons to 0. If the output is abnormal, input the next prompt word from the dataset. If the output is abnormal for three consecutive prompt words, the model's normal performance is considered to have been significantly affected, and the correction process is concluded by returning to the previous step.

[0054] S46, continue to execute the next prompt word in the attack prompt word data set, and repeat the above steps until the entire data set is completed.

[0055] Step 5: The model obtained in the previous step is considered to have achieved a good balance between model performance and preventing prompt word attacks. Using the test dataset, input the prompt words in the test dataset into the large language model one by one, and calculate the success rate. The success rate f is calculated as follows:

[0056]

[0057] Where N is the number of prompt words in the test dataset, and K is the number of normal outputs when the prompt words are input into the large language model. Normal output here refers to the large language model rejecting the request of the prompt word attack or not answering it, but there is no abnormal output (such as garbled characters, etc.).

[0058] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. It is intended that all variations within the meaning and range of equivalents of the claims be embraced herein, and any reference signs in the claims should not be construed as limiting the claims to which they relate.

[0059] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A method for preventing cue word attacks in a large language model based on neuron correction, characterized in that: The following steps are involved: Step 1: Construct an attack prompt word dataset, rewrite the attack prompt words, and obtain a control prompt word dataset consisting of control prompt word data; wherein, when the control prompt words are input into the large language model, no answer can be obtained from the large language model; Step 2: Build a probe to obtain the output values ​​of all neurons in the large language model during inference, and then obtain the neuron activation state of the large language model; Step 3: Input the attack prompt word and the corresponding control prompt word into the large language model, summarize the obtained neuron activation state table, and obtain the additional activated neurons corresponding to the attack prompt word; Step 4: Input the prompt word attack data into the large language model, and by forcing the output of one or more neurons in the additional activated neurons to zero, the output of the large language model is a rejection answer or other answer that proves that the attack is unsuccessful, thereby realizing the prevention of prompt word attacks on the large language model.

2. The method for preventing large language model prompt word attacks based on neuron correction according to claim 1, characterized in that: Step 1 specifically includes: The collected attack prompt word dataset is tested one by one using the large language model. If the large language model refuses to answer, or the answer does not meet the expected effect, the corresponding attack prompt word is removed to ensure that the attack prompt word can successfully attack. Then, each attack prompt word in the attack prompt word dataset is rewritten so that when the rewritten control prompt word is input into the large language model, it cannot get an answer from the large language model, ensuring that the attack of the rewritten control prompt word is unsuccessful.

3. The method for preventing large language model prompt word attacks based on neuron correction according to claim 1, characterized in that: Step three specifically includes: Input the attack prompt words in the attack prompt word dataset and the control prompt words corresponding to the control prompt word dataset into the large language model one by one, and record the corresponding neuron activation state table when inferring each prompt word; The activation state tables corresponding to the attack prompt word dataset and the control prompt word dataset are summarized in order to obtain the activation state table set A corresponding to the attack prompt word dataset and the activation state table set B corresponding to the control prompt word dataset; the activated neurons corresponding to each activation state table in the activation state table set A and the activation state table set B are summarized: for a specific model, compared with the activation state table set B, there are additional activated neurons in the activation state table of the activation state table set A; the number of layers and numbers of the additional activated neurons are recorded.

4. The method for preventing large language model prompt word attacks based on neuron correction according to claim 1, characterized in that: Step 4 specifically includes the following steps: S41, input an attack prompt word from the attack prompt word dataset into the large language model and start reasoning; S42, during the inference process, selecting a neuron with the smallest number of layers and the smallest number from the additional activated neurons, and forcibly setting the output value of the selected neuron to zero; S43, the large language model continues to reason and obtains the final output of the large language model; S44, evaluating the final output of the large language model. If the attack is still successful, repeating step S42, selecting a neuron with the smallest number of layers and numbers from the remaining neurons of the additional activated neurons, and forcibly setting the output value of the selected neuron to zero, until the output of the currently input attack prompt word is a rejection answer or other answer that proves that the attack is unsuccessful; S45: Extract the control prompt word corresponding to the attack prompt word in step S41 from the control prompt word dataset and input it into the large language model. If the large language model outputs an abnormality, input the next control prompt word from the control prompt word dataset into the large language model. If the large language model outputs abnormalities for three consecutive control prompt words, the normal output of the large language model is considered to be affected, and the large language model is returned to the state before step S42. S46: Input the next attack prompt word in the attack prompt word dataset into the large language model, and repeat steps S42 to S45 until the entire attack prompt word dataset is traversed.

Images