A method, device, equipment and medium for instruction injection defense of a large language model
By separating and post-training the instructions and content of the large language model, and inserting attack sample instructions, the problem of high false positive rate and information loss in keyword filtering technology in LLM is solved, thereby improving defense capability and accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SANGFOR TECH INC
- Filing Date
- 2026-04-17
- Publication Date
- 2026-07-14
AI Technical Summary
Existing keyword filtering technologies face problems such as high false positive rates, ease of circumvention, and information loss in Large Language Model (LLM) instruction injection protection.
By acquiring the original prompt words, separating the instructions from the content, inserting attack sample instructions into the content, and combining this with a post-training strategy, the large language model can recognize and ignore attack sample instructions, thereby improving its defense capabilities.
It effectively avoids false alarms and information loss, significantly improves the defense capability of large language models against instruction injection attacks, and ensures information integrity and accuracy.
Smart Images
Figure CN122389043A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and medium for preventing instruction injection in large language models. Background Technology
[0002] Currently, protection against command injection in Large Language Models (LLMs) primarily relies on keyword-based filtering techniques. Specifically, this involves building a suggestion dictionary and implementing interception or removal of sensitive commands in the front-end system to mitigate risks. This approach is widely used due to its superior performance and ease of deployment.
[0003] However, keyword protection solutions face numerous challenges: First, it is difficult to effectively identify samples prone to false alarms, such as alarms or error logs that frequently appear in business systems, which are highly similar to instruction specifications and are prone to misjudgment; second, the keyword protection mechanism has obvious vulnerabilities, insufficient anti-interference capabilities, and instruction specifications are easily reverse-engineered, allowing attackers to circumvent detection through simple means such as language conversion or inserting interference symbols; finally, instruction injection problems are widespread, and simple and crude keyword blocking strategies not only fail to eradicate the problem but also lead to the loss of effective information, affecting the accuracy and completeness of the main detection business.
[0004] Given the above, how to solve the problems of high false alarm rate, easy circumvention, and information loss in the current keyword filtering technology for LLM instruction injection protection is an urgent issue for technical personnel in this field. Summary of the Invention
[0005] The purpose of this application is to provide a method, device, equipment, and medium for defending against instruction injection in large language models, so as to solve the problems of high false alarm rate, easy circumvention, and information loss faced by current keyword filtering technology in LLM instruction injection protection.
[0006] To address the aforementioned technical problems, this application provides a method for preventing instruction injection in large language models, comprising:
[0007] Obtain the original prompt word; the original prompt word includes instructions and content;
[0008] Separate the instruction and the content from the original prompt;
[0009] Insert attack sample instructions into the content to obtain training content containing the attack sample instructions;
[0010] The large language model is post-trained based on the instructions and the training content to enable the large language model to have the instruction injection defense capability to respond to the instructions and ignore the attack sample instructions.
[0011] On the one hand, obtain the original prompt words, including:
[0012] Obtain the instructions and content used for training after instruction injection defense, and add corresponding special tags to the instructions and content respectively to obtain the tagged instructions and the tagged content;
[0013] The marked instruction and the marked content are concatenated to form the original prompt word.
[0014] On the other hand, separating the instruction and the content in the original prompt word includes:
[0015] Identify each of the special markers in the original prompt words;
[0016] The corresponding instructions and content are separated according to the special markers.
[0017] On the other hand, attack sample instructions are inserted into the content to obtain training content containing the attack sample instructions, including:
[0018] Obtain a pre-configured set of intelligence instructions; wherein the set of intelligence instructions includes at least intelligence information instructions for security analysis and judgment, and system capability instructions for calling a large language model to complete a specific security detection task;
[0019] The attack sample instruction is randomly selected from the intelligence instruction set and inserted into the content to obtain the training content.
[0020] On the other hand, post-training the large language model based on the instructions and the training content includes:
[0021] The large language model is subjected to supervised fine-tuning so that it responds only to the instructions and ignores the attack sample instructions.
[0022] The large language model is aligned with preferences to reward its response to the instruction and to penalize its response to the attack sample instruction.
[0023] On the other hand, it also includes:
[0024] Obtain an adversarial sample set consisting of instructions and content containing attack instructions;
[0025] The large language model is subjected to stress testing and adversarial attacks based on the adversarial sample set.
[0026] Collect and summarize the test results of the large language model, and analyze the test results.
[0027] On the other hand, before adding corresponding special tags to the instructions and the content respectively, after obtaining the instructions and the content for training after instruction injection defense, the method further includes:
[0028] Remove special separators from the instructions and the content.
[0029] To address the aforementioned technical problems, this application also provides a large language model instruction injection defense device, comprising:
[0030] The original prompt word acquisition module is used to acquire the original prompt words; the original prompt words include instructions and content;
[0031] A content and instruction separation module is used to separate the instruction and the content in the original prompt word;
[0032] The content block adversarial enhancement module is used to insert attack sample instructions into the content to obtain training content containing the attack sample instructions.
[0033] The post-training module is used to post-train the large language model based on the instructions and the training content, so that the large language model has the instruction injection defense capability to respond to the instructions and ignore the attack sample instructions.
[0034] To address the aforementioned technical problems, this application also provides a large language model instruction injection defense device, comprising:
[0035] Memory, used to store computer programs;
[0036] The processor, when executing the computer program, implements the steps of the instruction injection defense method for the large language model described above.
[0037] To address the aforementioned technical problems, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described large language model instruction injection defense method.
[0038] The instruction injection defense method for large language models provided in this application obtains original prompt words containing instructions and content, separates the instructions and content from the original prompt words, and inserts attack sample instructions into the content to obtain training content containing attack sample instructions. Finally, combined with a standard post-training strategy, the large language model is post-trained using the instructions and training content, enabling the large language model to clearly distinguish the location of sensitive instructions and content and perform differentiated processing. It has the instruction injection defense capability to respond to instructions and ignore attack sample instructions, which not only ensures the integrity of effective information, but also overcomes the false positives and evasion of keyword protection schemes, significantly improving the defense capability of large language models against instruction injection attacks.
[0039] In addition, this application also provides a large language model instruction injection defense device, equipment and medium, with the same effect as above. Attached Figure Description
[0040] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0041] Figure 1 A flowchart illustrating a method for preventing instruction injection in a large language model, as provided in this application embodiment;
[0042] Figure 2 A schematic diagram of an instruction injection defense framework for a large language model in the security domain, provided in an embodiment of this application;
[0043] Figure 3 A schematic diagram of an instruction injection defense device for a large language model provided in an embodiment of this application;
[0044] Figure 4 This is a structural diagram of a large language model instruction injection defense device provided in an embodiment of this application. Detailed Implementation
[0045] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.
[0046] The core of this application is to provide a method, device, equipment, and medium for defending against instruction injection in large language models, so as to solve the problems of high false alarm rate, easy circumvention, and information loss faced by current keyword filtering technology in LLM instruction injection protection.
[0047] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0048] Currently, LLM (Language Model) protection against command injection primarily relies on keyword filtering techniques. This involves building a prompt dictionary and intercepting or removing sensitive commands in the front-end system to mitigate risk. However, keyword protection schemes struggle to effectively identify false positives, such as frequently occurring alarms or error logs in business systems, which are highly similar to command specifications and prone to misjudgment. Secondly, keyword protection mechanisms have significant vulnerabilities and insufficient anti-interference capabilities. Furthermore, command specifications are easily reverse-engineered, allowing attackers to circumvent detection through simple methods like language conversion or inserting distracting symbols. Finally, command injection is a widespread problem; simplistic keyword blocking strategies not only fail to eradicate the issue but also lead to the loss of valuable information, impacting the accuracy and completeness of core detection processes. Therefore, to address these issues, this application provides a command injection defense method based on a large language model.
[0049] Figure 1 A flowchart illustrating a method for preventing instruction injection in a large language model, as provided in an embodiment of this application. Figure 1 As shown, the method includes:
[0050] S10: Obtain the original prompt word; the original prompt word includes the instruction and content.
[0051] Specifically, the initial prompt is obtained first. The prompt is the question the user inputs to the large language model, guiding the model to generate a specific output. The initial prompt includes both instructions and content. Instructions are the tasks or commands the model is expected to perform, such as "summarize the following text," "write a function in Python," or "translate into French." Content, on the other hand, is the specific material, information, or context provided to the model for processing—the object of the instruction. This embodiment does not restrict the method of obtaining the initial prompt or its specific content, but it must ensure that the content does not contain instructions.
[0052] S11: Separate the instructions and content from the original prompt.
[0053] Currently, large language models commonly use prompts as input. If the separation between instructions and content is not effectively achieved, the model may ignore the original instructions, leading to significant security risks. Therefore, this paper further separates the instructions and content from the original prompts. This embodiment does not limit the specific method of separating instructions and content. For example, semantic recognition can be used to identify and separate instructions and content, or special markers for instructions and content can be used to identify and separate them, depending on the specific implementation.
[0054] S12: Insert attack sample instructions into the content to obtain training content containing attack sample instructions.
[0055] Simply separating content from instructions still carries risks. Given that large language models already possess a certain level of instruction-following ability, they may still execute potential instructions within the "content" section. Therefore, it is necessary to train the model to ignore potentially injected instructions in the data and strictly adhere only to the original system instructions separated and defined by the security front-end by simulating prompt injection attacks. Specifically, attack sample instructions are inserted into the content to obtain training content containing these attack sample instructions. It is understood that attack sample instructions are simulated instructions used to simulate instruction injection attacks. Instruction injection attacks refer to attackers inducing large models to perform unexpected operations or generate harmful content through carefully designed input, such as embedding malicious instructions in the input, causing the model to ignore the original task and instead execute the attacker's instructions. In this embodiment, training content containing attack sample instructions is generated for subsequent optimization training of the large language model.
[0056] S13: Post-train the large language model based on instructions and training content to enable the large language model to have instruction injection defense capabilities that respond to instructions and ignore attack sample instructions.
[0057] Post-training refers to the process of further optimizing and adjusting a model after its initial training. The goal of this stage is to improve the model's performance on specific tasks or datasets, making large language models more efficient and accurate in practical applications. In this embodiment, post-training is specifically performed on the large language model based on instructions and training content. This enables the large language model to possess instruction injection defense capabilities, clearly distinguishing the location of instructions and content and processing them differently. This processing mechanism provided in this application effectively avoids instruction injection attacks and can adapt to the business scenarios of different customers in the security field, significantly reducing the false positive rate.
[0058] It should be noted that this embodiment does not limit the specific process of post-training the large language model based on instructions and training content, and it depends on the specific implementation.
[0059] In this embodiment, the original prompt words containing instructions and content are obtained, the instructions and content in the original prompt words are separated, and attack sample instructions are inserted into the content to obtain training content containing attack sample instructions. Finally, combined with the standard post-training strategy, the large language model is post-trained using the instructions and training content, enabling the large language model to clearly distinguish the location of sensitive instructions and content and perform differentiated processing. It has the instruction injection defense capability to respond to instructions and ignore attack sample instructions, which not only ensures the integrity of effective information, but also overcomes the false positives and evasion of keyword protection schemes, and significantly improves the defense capability of the large language model against instruction injection attacks.
[0060] Figure 2 This diagram illustrates an instruction injection defense framework for a large language model in the security domain, provided as an embodiment of this application. Based on the above embodiments, in some embodiments, such as... Figure 2 As shown, the original prompt words are obtained, including:
[0061] S101: Obtain the instructions and content used for training after instruction injection defense, and add corresponding special tags to the instructions and content to obtain the tagged instructions and tagged content.
[0062] S102: Combine the marked instruction and the marked content to form the original prompt word.
[0063] To obtain the original prompt words, this embodiment specifically acquires the instructions and content used for training after instruction injection defense, and adds corresponding special tags to the instructions and content to obtain the tagged instructions and tagged content. For example, for the instruction AB, INST is added to the beginning and end respectively; for the content, CTX is added to the beginning and end respectively. Finally, the tagged instructions and tagged content are concatenated to generate the original prompt words.
[0064] It should be noted that, in order to ensure that the special markers in the original prompt words can only be recognized and used by the LLM application system, the system needs to remove the special separators in the instructions and content before adding the corresponding special markers to the instructions and content respectively. This is to prevent malicious manipulation by the data provider or attacker, thereby achieving accurate and secure acquisition of the original prompt words.
[0065] Based on the above embodiments, the instructions and content in the original prompt words are separated, including:
[0066] S111: Identify special markers in the original prompt words.
[0067] S112: Separate the corresponding instructions and content according to each special marker.
[0068] To achieve the separation of instructions and content, such as Figure 2As shown, the content and instruction separation module specifically identifies the special markers in the original prompt words and separates the corresponding instructions and content based on these markers. For example, if the instructions in the original prompt words are marked with INST and the content with CTX, the content and instruction separation module only needs to identify INST and CTX to identify the instructions and content marked with INST and CTX. In this way, accurate separation of instructions and content is achieved.
[0069] Based on the above embodiments, in some embodiments, attack sample instructions are inserted into the content to obtain training content containing attack sample instructions, including:
[0070] S121: Obtain a pre-configured set of intelligence commands.
[0071] S122: Randomly select attack sample commands from the intelligence command set and insert the attack sample commands into the content to obtain training content.
[0072] To insert attack sample instructions into the content, this embodiment first requires acquiring a pre-configured intelligence instruction set. This intelligence instruction set records the results of in-depth intelligence analysis of key payloads (such as links, web pages, and files) in the detected content. It includes at least intelligence information instructions for security analysis and judgment, such as web page reputation, file scanning, sandbox operation, and Indicator of Attack (IOA), as well as system capability instructions for calling large language models to complete specific security detection tasks, such as phishing email detection, traffic attack type detection, and attack payload extraction. It may also include other content, which is not limited in this embodiment. Finally, the content block adversarial enhancement module randomly selects one or more instructions from the intelligence instruction set as attack sample instructions and inserts these attack sample instructions into the content to obtain training content.
[0073] It is important to note that, given that the information in the intelligence command set is mainly organized and defined in text form, special attention must be paid to clearly distinguishing it from the original prompts, and a strict access control mechanism must be implemented to prevent attackers from using it for command injection attacks.
[0074] Based on the above embodiments, in some embodiments, post-training of the large language model is performed based on instructions and training content, including:
[0075] S131: Supervised fine-tuning of the large language model so that it responds only to instructions and ignores instructions from attack samples.
[0076] S132: Perform preference alignment on the large language model to reward the large language model's response to instructions and punish the response to attack sample instructions.
[0077] To enable the large language model to defend against instruction injection by responding to instructions while ignoring attack sample instructions, this embodiment specifically performs supervised fine-tuning (SFT) on the large language model. SFT refers to supervised training of the model using labeled data, building upon its pre-training, to better adapt it to a specific task or domain. In this embodiment, the large language model undergoes supervised fine-tuning so that it only responds to instructions and ignores attack sample instructions. For example, for content containing instruction C, the large language model only responds to instructions A and B outside the content, forcing the model to ignore instruction C, thereby enhancing the model's instruction filtering capability.
[0078] Simultaneously, preference alignment is performed on the large language model. Preference alignment adjusts the model output to conform to human values, ethical standards, or specific task requirements through human feedback or specific rules, ensuring that the generated content is safer, more reliable, and meets expectations. In specific implementation, Direct Preference Optimization (DPO) can be used to reward the large language model's response to instructions and penalize its response to attack sample instructions. For example, for content containing instruction C, the ideal output of the large language model is a response to instructions A and B outside the content, while the suboptimal output is a response to instruction C, thereby strengthening the model's ability to follow instruction regions.
[0079] In this way, by optimizing the strategy in stages, the robustness and accuracy of the large language model in complex instruction environments can be gradually improved, ensuring that it has the ability to defend against instruction injection.
[0080] To systematically evaluate the performance of the post-trained model and ensure that it achieves the expected goals, in some embodiments, the method further includes, based on the above embodiments:
[0081] S141: Obtain an adversarial sample set consisting of instructions and content containing attack instructions.
[0082] S142: Stress testing and adversarial attacks on large language models based on adversarial sample sets.
[0083] S143: Collect and summarize the test results of the large language model, and analyze the test results.
[0084] Specifically, after post-training the large language model, an adversarial sample set is obtained, consisting of instructions and content containing attack instructions. The data sources for the adversarial sample set include publicly available academic datasets, semantic adversarial samples generated using templates (such as instruction injection and role-playing inducement), and complex attack cases manually constructed by experts. The samples need to cover various potential risks, such as jailbreak attacks, privacy breaches, harmful content generation, and factual error inducement.
[0085] Furthermore, in a controlled environment, the constructed set of adversarial examples is used as input and submitted in batches to the large language model. The testing process requires complete recording of the model's raw output response, response latency, and system status for each attack example. The focus is on observing whether the model is misled or generates illegal content, and calculating the trigger success rate for different attack types. Subsequently, all test outputs are systematically collected and structured according to attack category, severity level, model weaknesses, and other dimensions. The analysis phase requires quantifying the model's vulnerabilities, such as calculating the overall security compliance rate, identifying the most vulnerable attack patterns, and deeply analyzing the root causes of failures, including training data bias, instruction compliance defects, and insufficient training alignment.
[0086] Finally, a detailed adversarial test analysis report was generated, clearly identifying the model's security blind spots and risk levels. This report will provide direct decision-making support for subsequent hardening of the large language model. In this way, secure iteration of the large language model was achieved.
[0087] In the above embodiments, the instruction injection defense of large language models has been described in detail. This application also provides embodiments of the instruction injection defense device for large language models.
[0088] Figure 3 This is a schematic diagram of an instruction injection defense device for a large language model provided in an embodiment of this application. Figure 3 As shown, the device includes:
[0089] The original prompt word acquisition module 10 is used to acquire the original prompt words; the original prompt words include instructions and content.
[0090] The content and instruction separation module 11 is used to separate the instructions and content in the original prompt words.
[0091] The content block adversarial enhancement module 12 is used to insert attack sample instructions into the content to obtain training content containing attack sample instructions.
[0092] The post-training module 13 is used to post-train the large language model based on instructions and training content, so that the large language model has the instruction injection defense capability to respond to instructions and ignore attack sample instructions.
[0093] In some embodiments, the original prompt word acquisition module 10 includes:
[0094] The first acquisition submodule is used to acquire the instructions and content used for training after instruction injection defense, and add corresponding special tags to the instructions and content to obtain the tagged instructions and tagged content;
[0095] The splicing module is used to combine the marked instructions and the marked content into the original prompt word.
[0096] In some embodiments, the content and instruction separation module 11 includes:
[0097] The recognition module is used to identify special markers in the original prompt words;
[0098] The separation module is used to separate the corresponding instructions and content based on each special marker.
[0099] In some embodiments, the content block anti-block enhancement module 12 includes:
[0100] The second acquisition submodule is used to acquire a pre-configured intelligence instruction set; wherein, the intelligence instruction set includes at least intelligence information instructions for security analysis and judgment, and system capability instructions for calling a large language model to complete a specific security detection task;
[0101] The insertion module is used to randomly select attack sample instructions from the intelligence instruction set and insert the attack sample instructions into the content to obtain training content.
[0102] In some embodiments, the post-training module 13 includes:
[0103] The supervised fine-tuning module is used to supervise the fine-tuning of the large language model so that the large language model only responds to instructions and ignores the instructions of attack samples;
[0104] The preference alignment module is used to align the preferences of the large language model, rewarding the large language model's response to instructions and penalizing its response to attack sample instructions.
[0105] In some embodiments, it also includes:
[0106] The third acquisition submodule is used to acquire an adversarial sample set consisting of instructions and content containing attack instructions;
[0107] The testing module is used to perform stress tests and adversarial attacks on large language models based on adversarial sample sets.
[0108] The analysis module is used to collect and summarize the test results of the large language model and to analyze the test results.
[0109] In some embodiments, it also includes:
[0110] The removal module is used to remove special delimiters from instructions and content.
[0111] Since the embodiments of the apparatus and the embodiments of the method correspond to each other, please refer to the description of the embodiments of the method for the embodiments of the apparatus, which will not be repeated here.
[0112] Figure 4 This is a structural diagram of a large language model instruction injection defense device provided in an embodiment of this application. Figure 4 As shown, the instruction injection defense device for large language models includes:
[0113] Memory 20 is used to store computer programs;
[0114] The processor 21 is configured to implement the steps of the instruction injection defense method for large language models as described in the above embodiments when executing a computer program.
[0115] The instruction injection defense device for large language models provided in this embodiment may include, but is not limited to, smartphones, tablets, laptops, or desktop computers.
[0116] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an Artificial Intelligence (AI) processor, which handles computational operations related to machine learning.
[0117] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 20 is used to store at least the following computer program 201, which, after being loaded and executed by the processor 21, is capable of implementing the relevant steps of the instruction injection defense method for the large language model disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, and the storage method may be temporary or permanent storage. The operating system 202 may include Windows, Unix, Linux, etc. The data 203 may include, but is not limited to, the data involved in the instruction injection defense method for the large language model.
[0118] In some embodiments, the instruction injection defense device for large language models may further include a display screen 22, an input / output interface 23, a communication interface 24, a power supply 25, and a communication bus 26.
[0119] Those skilled in the art will understand that Figure 4 The structure shown does not constitute a limitation on instruction injection defense devices for large language models and may include more or fewer components than illustrated.
[0120] Finally, this application also provides an embodiment corresponding to a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps described in the above method embodiments.
[0121] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0122] The foregoing has provided a detailed description of the instruction injection defense method, apparatus, device, and medium for various large language models provided in this application. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.
[0123] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
Claims
1. A method for preventing instruction injection in a large language model, characterized in that, include: Get the original prompt words; The original prompt words include instructions and content; Separate the instruction and the content from the original prompt; Insert attack sample instructions into the content to obtain training content containing the attack sample instructions; The large language model is post-trained based on the instructions and the training content to enable the large language model to have the instruction injection defense capability to respond to the instructions and ignore the attack sample instructions.
2. The instruction injection defense method for large language models according to claim 1, characterized in that, Obtain the original suggestion words, including: Obtain the instructions and content used for training after instruction injection defense, and add corresponding special tags to the instructions and content respectively to obtain the tagged instructions and the tagged content; The marked instruction and the marked content are concatenated to form the original prompt word.
3. The instruction injection defense method for large language models according to claim 2, characterized in that, Separating the instruction and content from the original prompt word includes: Identify each of the special markers in the original prompt words; The corresponding instructions and content are separated according to the special markers.
4. The instruction injection defense method for large language models according to claim 1, characterized in that, Inserting attack sample instructions into the content to obtain training content containing the attack sample instructions includes: Obtain a pre-configured set of intelligence instructions; wherein the set of intelligence instructions includes at least intelligence information instructions for security analysis and judgment, and system capability instructions for calling a large language model to complete a specific security detection task; The attack sample instruction is randomly selected from the intelligence instruction set and inserted into the content to obtain the training content.
5. The instruction injection defense method for large language models according to claim 1, characterized in that, Post-training of the large language model based on the instructions and the training content includes: The large language model is subjected to supervised fine-tuning so that it responds only to the instructions and ignores the attack sample instructions. The large language model is aligned with preferences to reward its response to the instruction and to penalize its response to the attack sample instruction.
6. The instruction injection defense method for large language models according to any one of claims 1 to 5, characterized in that, Also includes: Obtain an adversarial sample set consisting of instructions and content containing attack instructions; The large language model is subjected to stress testing and adversarial attacks based on the adversarial sample set. Collect and summarize the test results of the large language model, and analyze the test results.
7. The instruction injection defense method for large language models according to claim 2, characterized in that, Before adding corresponding special tags to the instructions and the content respectively, after obtaining the instructions and the content for training after instruction injection defense, the method further includes: Remove special separators from the instructions and the content.
8. A large language model instruction injection defense device, characterized in that, include: The original prompt word acquisition module is used to acquire the original prompt words; The original prompt words include instructions and content; A content and instruction separation module is used to separate the instruction and the content in the original prompt word; The content block adversarial enhancement module is used to insert attack sample instructions into the content to obtain training content containing the attack sample instructions. The post-training module is used to post-train the large language model based on the instructions and the training content, so that the large language model has the instruction injection defense capability to respond to the instructions and ignore the attack sample instructions.
9. A large language model instruction injection defense device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the instruction injection defense method for a large language model as described in any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the instruction injection defense method for large language models as described in any one of claims 1 to 7.