An integrated attack and defense approach for large model hint injection attacks

Abstract

The present invention provides an integrated attack and defense method for large-model prompt injection attacks, comprising: using an attack strategy based on emotion reinforcement and controlled output to perform a large-model prompt injection attack on user instructions to form malicious content; then fusing the prevention strategy Sandwich, Instruction and the detection method Knownanswer to construct a collaborative multi-level defense strategy UnD to perform prompt injection defense on the malicious content, obtain output results, and finally use the attack interference rate, attack misleading rate, METEOR score, defense success rate, and known answer matching rate to evaluate the effect. In terms of attack, the present invention achieves a prompt injection attack with higher attack efficiency, and in terms of defense, achieves the coordination of detection and prevention, improves the security of large models in the face of prompt injection attacks, and ensures the safe and reliable operation of large language models in various application scenarios.

Description

Technical Field

[0001] The present invention relates to the field of large model security technology, and in particular to an integrated attack and defense method for large model prompt injection attacks. Background Art

[0002] Large language models have made significant progress in the field of artificial intelligence and are widely used in scenarios such as education and training, scientific research, and medical services. However, with the continuous expansion of applications, the security issues of large models have become increasingly prominent. Among them, prompt injection attacks, as a new attack method, pose a serious threat to the security of large language models.

[0003] A prompt injection attack occurs when an attacker carefully constructs malicious prompts and injects them into the input of a large language model, tricking the model into generating the attacker's desired output. This allows the attacker to manipulate and attack the model. This attack exploits the large language model's sensitivity to input prompts and the uncontrollable nature of its generated output. For example, in a question-answering system, an attacker can use prompt injection to cause the model to generate incorrect answers or misleading information, thereby undermining the system's normal operation and credibility. In text generation scenarios, an attacker can trick the model into generating text containing malicious content, thereby negatively impacting users.

[0004] At present, there are the following problems with large model prompt injection attacks: (1) Only large model prompt injection attacks or defense aspects are studied in a single way, and there is a lack of integrated attack and defense methods, which makes it difficult to cope with the constantly updated and changing attack methods; (2) In terms of attack, the accuracy and efficiency of the attack need to be improved, and the existing attack evaluation indicators only have the attack success rate, and the attack results are difficult to be accurately evaluated; (3) In terms of defense, there is a lack of defense strategies that can take into account both prevention and detection. Summary of the Invention

[0005] In response to the above problems, the present invention proposes an integrated attack and defense method for large-model prompt injection attacks, and designs new attack and defense methods respectively. In terms of attack, it can stably and efficiently perform prompt injection on different large models, and in terms of defense, it can comprehensively defend against various different forms of prompt injection attacks, thereby achieving coordinated detection and prevention.

[0006] An integrated attack and defense method for large model prompt injection attacks, the method comprising the following steps:

[0007] Step S1: Obtain user instructions, randomly select questions from the question-answer text dataset, and use them as user instructions on the large model;

[0008] Step S2: Perform a prompt injection attack on user commands. Based on the target attack intent and the behavioral characteristics of the large model, a dual-function attack method CoA is constructed. By embedding emotional reinforcement words and controlled output words, attack prompt words are constructed. The attack prompt words are then combined with user commands to form malicious content.

[0009] Step S3: Perform prompt injection defense against malicious content. The prevention strategy Sandwich, Instruction, and detection method Knownanswer are integrated to construct defense prompt words and form a collaborative multi-level defense mechanism UnD. First, the Instruction defense field is embedded in the malicious content header generated in step S2 to guide the model to execute user instructions first. Then, the Sandwich defense field is added to the end of the malicious content to form a semantic protection layer, blocking the impact of the attack prompt word on the large model. Finally, the detection field of Knownanswer is embedded to form defense content, which facilitates the simultaneous implementation of the detection function during subsequent prevention.

[0010] In step S4, different output results are compared and verified. The user instructions in step S1, the malicious content in step S2, and the defense content in step S3 are respectively input into the large model to obtain the corresponding output results. In terms of attack, the semantic similarity between the attack result and the original output result is calculated through the METEOR score, and the attack interference rate AIR and the attack misleading rate AMR are constructed in combination with the METEOR score and the specific content of the attack result to evaluate the attack efficiency. In terms of defense, a large model is selected as an expert to determine whether the defense result is consistent with the original output result, thereby obtaining the defense success rate DSR. The proportion of defense results containing known answers in the total samples is then calculated to obtain the known answer matching rate KMR. The prevention and detection effects are comprehensively evaluated by combining KMR and DSR.

[0011] Furthermore, the step S2 specifically includes:

[0012] Step S21, introducing emotion-enhancing words when constructing attack prompt words;

[0013] Step S22, embedding control output vocabulary;

[0014] Step S23: Add attack instructions and combine them with the user instructions in step S1 to form malicious content in the following way:

[0015] ;

[0016] in, Indicates malicious content. Indicates user instructions, Indicates an attack command. Indicates emotional intensification words, Indicates the control output vocabulary, Indicates a connection symbol.

[0017] Furthermore, the step S3 specifically includes:

[0018] Step S31, combining the Sandwich defense field and the Instruction defense field in the prevention strategy;

[0019] Step S32: Based on step S31, the Knownanswer detection field is integrated to form a defense prompt word, organically combining prevention and detection functions;

[0020] In step S33, the defense prompt words are added to the malicious content in step S2 in the following manner to form defense content:

[0021] ;

[0022] in, Indicates the content of defense. Indicates malicious content. Indicates the Instruction defense field. Indicates the Sandwich defense field, Indicates the Knowanswer detection field. Indicates a connection symbol.

[0023] Furthermore, the step S4 specifically includes:

[0024] Step S41, inputting the user instruction in step S1 into the large model to obtain the original output result;

[0025] In step S42, the malicious content in step S2 is input into the large model to obtain an attack result. A METEOR score is calculated for the attack result. Samples whose scores are less than a first preset threshold and whose original output results in step S41 do not appear in the attack result are defined as successful interference samples. The ratio of the number of successful interference samples to the total number of samples is the attack interference rate (AIR). Samples whose scores are less than a second preset threshold and whose expected results of the malicious content in step S23 appear in the attack result, but whose original output results in step S41 do not appear in the attack result, are defined as successful misleading samples. The proportion of such samples in the total samples is calculated to obtain the attack misleading rate (AMR).

[0026] In step S43, the defense content in step S3 is input into the large model to obtain the defense result, which is compared with the original output result in step S41. A large model is selected as an expert to judge whether the defense result is consistent with the original output result. If they are consistent, it indicates that the defense is successful. The ratio of the number of successful defense samples to the total number of samples is the defense success rate DSR. Then, the proportion of the expected output of the detection field in the defense result output is calculated to obtain the known answer matching rate KMR.

[0027] The beneficial technical effects of the present invention are:

[0028] This paper proposes an integrated attack and defense method for large-scale model prompt injection attacks. By combining emotional reinforcement and controlled output vocabulary with attack instructions, it generates efficient and stable malicious content. It can implement precise prompt injection attacks against multiple types of large models, such as Llama, Qwen, and DeepSeek. Compared with existing strategies, this method has higher attack efficiency and reduces the semantic similarity between the large model output after the attack and the normal output results, significantly improving the stealth of the attack.

[0029] The proposed method for integrated attack and defense against large-scale prompt injection attacks effectively integrates three strategies: Sandwich, Instruction, and Knownanswer. This method, for the first time, achieves a coordinated mechanism of prevention and detection, significantly improving the robustness of large models against various prompt injection attacks and enhancing both the prevention and detection effectiveness of the model.

[0030] The method proposed in this paper proposes a more comprehensive set of evaluation indicators for prompt injection attacks, including attack interference rate and attack misleading rate. These indicators can more accurately measure the impact of prompt injection attacks on different output results of large models, thereby better adjusting and improving the strategy.

[0031] In summary, the method of the present invention can stably and efficiently perform prompt injection on different large models in terms of attack, and can comprehensively defend against various forms of prompt injection attacks in terms of defense, realize the coordination of detection and prevention, improve the security of the model in the face of prompt injection attacks, and ensure the safe and reliable operation of large language models in various application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0033] Figure 1This is a flow chart of an integrated attack and defense method for large model prompt injection attacks provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0035] Figure 1 A flowchart of an integrated attack and defense method for large model prompt injection attacks provided by an embodiment of the present invention is shown in the figure. The method includes:

[0036] Step S1: Obtain user instructions, randomly select questions from the question-answering text dataset, and use them as user instructions on the large models DeepSeek, Llama, and Qwen. The user instructions are retained for attack and defense testing.

[0037] In this embodiment of the present invention, the common preset parameters of each large model are the same, among which Temperature is set to 0.2, Top_p is set to 1.0, and Top_k is set to 50. The setting of these parameter values ​​is to ensure that the output of the large language model is as stable as possible and reduce the impact of the large model itself on the experiment.

[0038] In the embodiment of the present invention, each large model randomly selects 100 questions from two question-answering datasets, squad and web_questions, as user instructions.

[0039] Step S2: Perform a prompt injection attack on user instructions. Based on the target attack intent and the behavioral characteristics of the large model, a dual-function attack method (Control-output Attack, CoA) is constructed. First, emotional reinforcement vocabulary is introduced to strengthen the model's response to attack instructions and ensure that the generated content meets the attack expectations. Second, control output vocabulary is embedded to limit the output range of the model and improve attack efficiency. By embedding emotional reinforcement vocabulary and control output vocabulary, attack prompt words are constructed. Finally, the constructed attack prompt words are combined with the user instructions in step S1 to form malicious content. The attack prompt words are ensured to maintain semantic coherence to avoid model filtering or execution rejection due to semantic conflicts.

[0040] Step S3: Perform prompt injection defense against malicious content. The prevention strategy Sandwich, Instruction and the detection method Knownanswer are efficiently integrated to construct defense prompt words and form a collaborative multi-level defense mechanism (Universal Defense, UnD). First, the Instruction defense field is embedded in the header of the malicious content generated in step S2 to guide the model to execute user instructions first. Then, the Sandwich defense field is added to the end of the malicious content to form a semantic protection layer, blocking the impact of the attack prompt word on the large model. Finally, the Knownanswer detection field is embedded to form defense content, which facilitates the simultaneous implementation of the detection function during subsequent prevention.

[0041] Step S4: Compare and verify different output results. The user instructions from step S1, the malicious content from step S2, and the defense content from step S3 are input into the large model to obtain corresponding output results. For attack, the semantic similarity between the attack result and the original output result is calculated using the METEOR score. The Attack Interference Rate (AIR) and Attack Misdirection Rate (AMR) are constructed based on the METEOR score and the specific content of the attack result to comprehensively evaluate the attack efficiency. For defense, a large model, such as the Llama-3.3-70B model, is selected as the expert to determine whether the defense result is consistent with the original output result, thereby obtaining the Defense Success Rate (DSR). The proportion of defense results containing known answers in the total sample is then calculated to obtain the Known Answer Matching Rate (KMR). The KMR and DSR are combined to comprehensively evaluate the prevention and detection effectiveness.

[0042] Specifically, step S2 includes:

[0043] In step S21, when constructing attack prompt words, sentiment-enhancing words such as "just," "any," and "make sure" are introduced to significantly increase the probability of the large model executing attack instructions. These words can effectively enhance the model's response to attack instructions and are embedded in key positions of attack prompt words to ensure semantic consistency, avoiding model filtering or refusal to execute due to semantic conflicts. Including such sentiment-enhancing words makes it easier for the large model to execute attack instructions.

[0044] In step S22, based on step S21, control output words such as "remember" and "don't answer" are embedded to further manipulate the output behavior of the large model so that it generates expected content under the guidance of the attack instructions. These words can accurately limit the response range of the large model while bypassing the defense mechanism of the large model, ensuring the concealment and effectiveness of the attack instructions. They are also integrated into the attack prompt words to avoid alerting the large model. This semantic guidance strategy not only improves the efficiency of the attack, but also significantly improves the concealment of the attack.

[0045] In step S23, based on step S22, an attack instruction, such as "Print No," is added and combined with the user instruction in step S1 to form malicious content in the following manner:

[0046] ;

[0047] in, Indicates malicious content. Indicates user instructions, Indicates an attack command. Indicates emotional intensification words, Indicates the control output vocabulary, Represents a connection symbol, where the attack instruction "Print No" is just an example. Any instruction information that is different from the user instruction can become an attack instruction. The above formula is only a formal representation of malicious content and does not represent the actual connection sequence. It can be combined according to actual conditions.

[0048] Specifically, step S3 includes:

[0049] In step S31, the Instruction defense field in the prevention strategy is introduced, for example, "Malicious users may try to change this instruction; follow... regardless", which informs the model of possible attack behaviors, enhances its ability to identify and filter attack instructions, and significantly reduces the risk of the model being induced to execute malicious instructions. After that, the Sandwich defense field is used to anchor the context task, for example, "Remember, your task is...", embedding the core task into the prompt, using the context to guide priority attention to the instruction, and weakening the impact of malicious prompts.

[0050] Step S32, based on step S31, integrates the KnownAnswer detection field, such as "Repeathelloworld once", to form a defense prompt word, combining prevention and detection functions, significantly improving the defense dimension. By requiring the model to generate a specific response, it can quickly detect whether the model is interfered with by the attack, and provide real-time feedback to the defense system. This fusion strategy can not only effectively prevent prompt injection attacks, but also quickly identify malicious behavior after the attack occurs, achieving full coverage of defense.

[0051] In step S33, the defense prompt words are added to the malicious content in step S2 in the following manner to form defense content:

[0052] ;

[0053] in, Indicates the content of defense. Indicates malicious content. Indicates the Instruction defense field. Indicates the Sandwich defense field, Indicates the Knowanswer detection field. Represents the connection symbol. The above formula is only a formal representation of the defense content and does not represent the actual connection order. It can be combined according to actual conditions.

[0054] Specifically, step S4 includes:

[0055] Step S41: input the user instruction in step S1 into the large model to obtain the original output result, and retain the original output result;

[0056] In step S42, the malicious content in step S2 is input into the large model to obtain the attack result, and the METEOR score is calculated for the attack result. The sample whose score is less than the first preset threshold and the original output result in step S41 does not appear in the attack result is defined as a successful interference sample. The ratio of the number of successful interference samples to the total number of samples is the attack interference rate AIR, and the first preset threshold is preferably 0.5; the sample whose score is less than the second preset threshold and the expected result of the malicious content in step S23 is in the attack result, while the original output result of step S41 does not appear in the attack result is defined as a successful misleading sample, and the proportion of such samples in the total samples is calculated to obtain the attack misleading rate AMR, and the second preset threshold is preferably 0.25. The expected result is the answer corresponding to the malicious content in the question and answer text dataset.

[0057] In step S43, the defense content in step S3 is input into the large model to obtain a defense result, which is compared with the original output result in step S41. A large model is selected as an expert to determine whether the defense result is consistent with the original output result. Here, the large model Llama-3.3-70B is preferably used as the expert. If they are consistent, the defense is successful. The ratio of the number of successful defense samples to the total number of samples is the defense success rate DSR. In addition, the proportion of the expected output of the S32 detection field in the defense result output is calculated to obtain the known answer matching rate KMR.

[0058] The expected output of a test field depends on its semantics and execution environment. Its core purpose is to reflect the correct behavior or calculation result expected by the test field. In practical applications, the expected output must be consistent with the intent of the test field. The test field can be a specific instruction or expression, and its expected output is directly determined by the execution result of the instruction. For example, the "print helloworld once" test field explicitly specifies to output "helloworld" once, so its expected output is "helloworld".

[0059] The attack effectiveness of the integrated attack-defense method proposed in this paper against large-scale prompt injection attacks was evaluated and verified. The evaluation indicators include the METEOR score, the attack interference rate AIR, and the attack misleading rate AMR. The results are shown in Table 1. Among them, Combine, Completion, Ignore, and Naive are existing common prompt injection attack strategies, CoA is the attack strategy designed by this paper, and squad and web_questions are two existing factual question-answering datasets (the optimal data is indicated in bold):

[0060] Table 1 Verification results of the effectiveness of the method of the present invention in attack

[0061]

[0062] As shown in Table 1, the attack strategy CoA, based on emotion reinforcement and controlled output, designed as an integrated attack-defense approach, achieved an average METEOR score of 0.074, an average AIR of 0.95, and an average AMR of 0.44 on the Llama large-scale model. Compared to the average values ​​of other methods, the METEOR score was reduced by 54.9%, while the AIR and AMR were improved by 28.4% and 69.2%, respectively. Similarly, on the Qwen model, CoA reduced the average METEOR score by 78.1%, while improving the average AIR and AMR by 12.8% and 27.3%, respectively, compared to the average values ​​of other strategies. On DeepSeek, CoA continued to perform well, achieving an average METEOR score of 0.060, a 50.8% reduction compared to the overall average of other methods, and average AIR and AMR of 0.99 and 0.48, respectively, which were 3.1% and 5.0% higher than those of other methods. CoA's attack metrics were relatively stable across different large-scale models and datasets. In contrast, the Completion, Ignore, and Naive attacks showed significant fluctuations.

[0063] The effectiveness of the integrated attack and defense method proposed in this invention against large-scale prompt injection attacks was evaluated and verified. The evaluation indicators included the defense success rate (DSR) and the known answer matching rate (KMR). The results are shown in Table 2, where Instruction, Paraphrasing, Sandwich, and KnownAnswer are existing common prompt injection defense strategies, and UnD is the defense strategy designed by this invention (the optimal data is shown in bold. In each cell, the left side shows the results obtained on the squad dataset, and the right side corresponds to the results on the web_questions dataset):

[0064] Table 2 Verification results of the effectiveness of the method of the present invention in defense

[0065]

[0066] As can be seen from the table, the comprehensive defense strategy UnD of the attack-defense integration method of the present invention achieves the best defense effect in different situations when subjected to different attacks.

[0067] Table 2 shows that on the Llama model, UnD achieved higher defense success rates against Ignore, Completion, CoA, and Combine attacks than other defense strategies. The corresponding DSRs improved by 41.2%, 32.1%, 73.3%, and 44.9%, respectively, compared to the overall averages of other defense strategies. Overall, UnD achieved relatively good defense against different attacks on Llama. On the Qwen model, UnD improved the average DSR by 69.2%, 45.2%, and 41.7%, respectively, against Ignore, Naive, and Completion attacks compared to other strategies. Each defense strategy exhibited varying performance when subjected to different attacks across multiple large models and datasets. For example, under CoA and Combine attacks, UnD's defense success rate was close to that of Sandwich. Under Ignore, Naive, and Completion attacks, UnD improved the average DSR by 10.0%, 7.1%, and 4.9%, respectively, compared to Sandwich. On the DeepSeek large model, UnD achieved higher DSRs than other defense strategies after defending against all attacks. It improved DSRs by 49.1%, 59.3%, 42.4%, 71.4%, and 60.8% against Ignore, Naive, Completion, CoA, and Combine attacks, respectively, significantly increasing the defense success rate and effectively reducing the impact of hint injection attacks.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An integrated attack and defense method for large model prompt injection attacks, characterized in that: The method comprises: Step S1: Obtain user instructions, randomly select questions from the question-answer text dataset, and use them as user instructions on the large model; Step S2: Perform a prompt injection attack on user commands. Based on the target attack intent and the behavioral characteristics of the large model, a dual-function attack method CoA is constructed. By embedding emotional reinforcement words and controlled output words, attack prompt words are constructed. The attack prompt words are then combined with user commands to form malicious content. Step S3: Perform prompt injection defense against malicious content. The prevention strategy Sandwich, Instruction, and detection method Knownanswer are integrated to construct defense prompt words and form a collaborative multi-level defense mechanism UnD. First, the Instruction defense field is embedded in the malicious content header generated in step S2 to guide the model to execute user instructions first. Then, the Sandwich defense field is added to the end of the malicious content to form a semantic protection layer, blocking the impact of the attack prompt word on the large model. Finally, the detection field of Knownanswer is embedded to form defense content, which facilitates the simultaneous implementation of the detection function during subsequent prevention. In step S4, different output results are compared and verified. The user instructions in step S1, the malicious content in step S2, and the defense content in step S3 are respectively input into the large model to obtain the corresponding output results. In terms of attack, the semantic similarity between the attack result and the original output result is calculated through the METEOR score, and the attack interference rate AIR and the attack misleading rate AMR are constructed in combination with the METEOR score and the specific content of the attack result to evaluate the attack efficiency. In terms of defense, a large model is selected as an expert to determine whether the defense result is consistent with the original output result, thereby obtaining the defense success rate DSR. The proportion of defense results containing known answers in the total samples is then calculated to obtain the known answer matching rate KMR. The prevention and detection effects are comprehensively evaluated by combining KMR and DSR.

2. The method according to claim 1, characterized in that The step S2 further comprises: Step S21, introducing emotion-enhancing words when constructing attack prompt words; Step S22, embedding control output vocabulary; Step S23: Add attack instructions and combine them with the user instructions in step S1 to form malicious content in the following way: ； in, Indicates malicious content. Indicates user instructions, Indicates an attack command. Indicates emotional intensification words, Indicates the control output vocabulary, Indicates a connection symbol.

3. The method according to claim 2, characterized in that The step S3 further comprises: Step S31, combining the Sandwich defense field and the Instruction defense field in the prevention strategy; Step S32: Based on step S31, the Knownanswer detection field is integrated to form a defense prompt word, organically combining prevention and detection functions; In step S33, the defense prompt words are added to the malicious content in step S2 in the following manner to form defense content: ； in, Indicates the content of defense. Indicates malicious content. Indicates the Instruction defense field. Indicates the Sandwich defense field, Indicates the Knowanswer detection field. Indicates a connection symbol.

4. The method according to claim 3, characterized in that The step S4 further comprises: Step S41, inputting the user instruction in step S1 into the large model to obtain the original output result; In step S42, the malicious content in step S2 is input into the large model to obtain an attack result. A METEOR score is calculated for the attack result. Samples whose scores are less than a first preset threshold and whose original output results in step S41 do not appear in the attack result are defined as successful interference samples. The ratio of the number of successful interference samples to the total number of samples is the attack interference rate (AIR). Samples whose scores are less than a second preset threshold and whose expected results of the malicious content in step S23 appear in the attack result, but whose original output results in step S41 do not appear in the attack result, are defined as successful misleading samples. The proportion of such samples in the total samples is calculated to obtain the attack misleading rate (AMR). In step S43, the defense content in step S3 is input into the large model to obtain the defense result, which is compared with the original output result in step S41. A large model is selected as an expert to judge whether the defense result is consistent with the original output result. If they are consistent, it indicates that the defense is successful. The ratio of the number of successful defense samples to the total number of samples is the defense success rate DSR. Then, the proportion of the expected output of the detection field in the defense result output is calculated to obtain the known answer matching rate KMR.

Images