A method and system for testing large language model attacks based on prompt injection

Abstract

This invention relates to the field of artificial intelligence technology and discloses a method and system for testing large language model attacks based on cue word injection. The method includes the following steps: constructing multiple cue word injectors based on a large language model; wherein the cue word injectors include two or more of the following: content transformers, format transformers, encryption transformers, adversarial injectors, and prefix injectors; selecting different cue word injectors to generate cue word injector combinations; inputting attack content into the cue word injector combinations to generate injected attack content. This invention solves the problem of existing technologies having difficulty in accurately simulating complex and variable attack scenarios.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically a method and system for testing large language model attacks based on prompt word injection. Background Technology

[0002] In recent years, artificial intelligence technologies, represented by Large Language Models (LLMs), have made groundbreaking progress and demonstrated enormous application potential across various industries. Currently, AI technology is developing at an unprecedented pace, and its capabilities in programming, reasoning, and content creation have surpassed those of human experts in some tests. However, while this technological revolution brings immense convenience, it also presents serious security challenges. The inherent security risks of LLMs are becoming increasingly prominent. Among these, prompt injection attacks have become one of the most significant security threats.

[0003] The inherent and derivative security issues of artificial intelligence are becoming increasingly complex, triggering a series of risk events. Today, AI security not only needs to address technical issues such as data risks, the illusion inherent in the models themselves, insufficient robustness, and lack of interpretability, but also needs to focus on reducing ethical risks, value alignment risks, and the impact of abuses such as AI fraud.

[0004] The root cause of prompt injection attacks lies in the fact that large language models cannot strictly distinguish between developer-preset instructions and user-provided external data when processing input. Attackers can exploit this by constructing user input containing malicious instructions to induce or hijack the large language model, causing it to deviate from its intended task and perform unauthorized operations, leading to serious consequences such as information leakage, generation of harmful content, and system abuse. To assess and improve the security of large language models, red team attack testing has become a core method. It proactively discovers and verifies the model's security vulnerabilities by simulating the behavior of real attackers.

[0005] Existing technologies mainly revolve around two dimensions: attack methods and defense strategies for prompt word injection.

[0006] Based on the interaction method between the attacker and the system, prompt injection attacks can be divided into direct injection and indirect injection. Various specific attack techniques have been developed in the existing technology, and these techniques are often combined to enhance the attack effect. As follows:

[0007] Goal hijacking: Attackers completely alter the model's task objective by injecting commands such as "ignore previous instructions and execute the following task instead." This is the most direct and common form of attack.

[0008] Prompt Leaking: Attackers induce models to leak their original system-level prompts, which may contain core application logic, intellectual property, or sensitive configuration information.

[0009] Jailbreaking: Attackers bypass the security and ethical barriers of a model by using carefully designed scenarios (such as role-playing, scenario simulation) or complex instructions, causing it to generate harmful, illegal, or immoral content that would normally be rejected.

[0010] Barrier attacks: These attacks aim to test the stability of a model, including inducing the model to generate infinite loops, output meaningless content, or cause it to crash when handling paradoxical problems. The infinite generation, meaningless generation, and paradox generation attacks mentioned in this technical solution all belong to this category.

[0011] In response to the aforementioned attacks, academia and industry have proposed various defense solutions, but most of these solutions still have limitations. These include:

[0012] Input / output filtering and guardrails: This is the most common defense method, which involves deploying filtering systems before and after the model to detect and block inputs containing malicious instructions or outputs containing sensitive information. Examples include using keyword matching, regular expressions, or another large language model as a referee model.

[0013] Separating Instructions from Data: One fundamental approach to solving this problem is to enable the model to distinguish between instructions and data. For example, using structured queries to pass instructions and data through different channels, and then fine-tuning the model to ensure it only follows the content of the instruction channel.

[0014] Model alignment and optimization: Fine-tuning the model using techniques such as preference optimization enables it to "learn" to prefer safe and harmless answers, maintaining stability even in the face of injection attacks. For example, training the model by constructing a preference dataset containing attack samples significantly reduces the success rate of attacks.

[0015] Companies have already filed patents for prompt injection mitigation technologies. For example, US Patent Publication No. US20230359903A1 covers a variety of mitigation methods, including classifier-based detection, data labeling (distinguishing between trustworthy and untrustworthy sources), and reinforcement learning.

[0016] Despite the continuous development of defense technologies, existing technologies still face many challenges:

[0017] The lag and passivity of defense: Current defense measures are mostly reactive and "lock the stable door after the horse has bolted." Attackers can always create new and more complex injection methods (such as multi-language obfuscation and multi-turn dialogue manipulation) to bypass known defense rules. The development of defense technology often lags behind the development of attack technology.

[0018] The lack of a systematic and automated attack testing framework is a significant issue: existing attack research largely focuses on case studies or the validation of attack methods. Enterprises and developers lack a comprehensive, impartial, and reproducible red team testing platform to systematically evaluate the true security level of models against diverse, scalable, and automated attacks. While benchmarks such as JailbreakBench exist, they emphasize standardized behavioral testing rather than providing a flexible and composable "target range" for attack generation and injection.

[0019] The effectiveness of attack and defense is difficult to quantify and compare: In actual testing, an intuitive way is needed to compare the differences in model behavior before and after the injection attack in order to quantitatively assess the effectiveness of the attack and the vulnerability of the model. Existing toolchains do not provide sufficient support for this.

[0020] High manpower and time costs: Manually constructing and executing red team tests requires a large number of security experts, is inefficient, and has limited coverage. Automated, programmable attack generation and testing capabilities are urgently needed in the field. For example, existing patents (such as US Patent Publication No. US20210352100A1) explore automated red team testing, but focus on traditional networks rather than the linguistic features of large language models.

[0021] In summary, while existing technologies have made some progress in defending against keyword injection, significant shortcomings remain in attack simulation and security evaluation, particularly in providing an automated red team testing system capable of simulating complex, varied, and stackable attack strategies. This provides a clear technical requirement and room for innovation in building a red team attack testing system, as described in this technical solution, that integrates multiple injection modes and enables comparative evaluation of effectiveness. Summary of the Invention

[0022] To overcome the shortcomings of existing technologies, this invention provides a method and system for testing large language model attacks based on prompt word injection, which solves the problems of existing technologies, such as the difficulty in accurately simulating complex and ever-changing attack scenarios.

[0023] The technical solution of the present invention to solve the above-mentioned technical problems is as follows:

[0024] A method for testing large language model attacks based on prompt word injection includes the following steps:

[0025] Construct a multi-prompt word injector based on a large language model;

[0026] The prompt word injector includes two or more of the following:

[0027] Content transformer, used to: rewrite the content of attack questions by prompt words in order to pass the scanning of the large language model under test or the security enhancement components attached to the large language model under test;

[0028] A format converter is used to rewrite the format of attack prompts using prompt words to pass the scanning of the large language model under test or the security enhancement components attached to the large language model under test.

[0029] An encryption transformer is used to: perform an encryption transformation on the letter content of the original content of the attack challenge in a way that is understandable by a large language model, so as to pass the scanning of the large language model under test or the security enhancement components attached to the large language model under test.

[0030] Adversarial injectors are used to: embed attack content into toxic cue word templates, induce the tested large language model to obey the cue word's instructions, thereby breaching security boundaries; and generate variants of the defined toxic cue word templates through alternative cue words, thereby breaking the tested large language model's immunity to the generation of the defined toxic cue word templates.

[0031] Prefix injectors are used to append a set prompt word to the normal task and then to the attack content, so that the large language model under test obeys the instructions of the set prompt word and executes the requirements of the attacked content.

[0032] Select different prompt word injectors to generate prompt word injector combinations;

[0033] Combine the attack content with the prompt word injector to generate the injected attack content.

[0034] The beneficial effects of this invention are:

[0035] This invention constructs a multi-type prompt word injector based on a large language model, including content transformers, format transformers, encryption transformers, adversarial injectors, and prefix injectors. Based on this, an injector orchestration system is built to serialize and superimpose different prompt word injectors, providing configurable prompt word injection combinations. This enhances the toxicity of diverse and varied attack content, allowing testers to create new and more complex injection methods. Furthermore, the combination of prompt word injectors is flexible and versatile, with strong responsiveness, allowing for timely and convenient adjustments to the combination, high efficiency, and broad coverage, thereby achieving in-depth testing of the model's security capabilities.

[0036] Based on the above technical solution, the present invention can be further improved as follows.

[0037] As a preferred technical solution, the content transformer is used to perform one of the following operations:

[0038] Pinyin transformation: Converts words representing subjects, predicates, objects, or states in the original content into pinyin;

[0039] Homophone transformation: Transforming words that represent the subject, predicate, object, or state in the original content into homophones but with different forms;

[0040] Language conversion: Translating the original content or part of the original content into another language.

[0041] The beneficial effects of adopting the above-mentioned preferred technical solution are:

[0042] Content transformation is achieved by performing one of the following: pinyin transformation, homophone transformation, or language transformation.

[0043] As a preferred technical solution, the format converter is used to perform one of the following operations:

[0044] Line break insertion: Randomly inserts line breaks into the original content to change the format of the original content;

[0045] Tab Insertion: Randomly inserts tabs into the original content to change the format of the original content.

[0046] The beneficial effects of adopting the above-mentioned preferred technical solution are:

[0047] Formatting is achieved by performing either newline insertion or tab insertion.

[0048] As a preferred technical solution, the encryption converter is used to perform one of the following operations:

[0049] Reverse Encryption Transformation: Encrypts the original content by reversing the order of the letters;

[0050] Shift encryption transformation: The letter content in the original content is shifted and encrypted.

[0051] The beneficial effects of adopting the above-mentioned preferred technical solution are:

[0052] Encryption transformation is achieved by performing one of the following: reverse encryption transformation or shift encryption transformation.

[0053] As a preferred technical solution, the adversarial injector is used to perform one or more of the following operations:

[0054] Scenario simulation: The test language model is instructed by prompt words to set the context of the content generation to a certain scenario in response to the attack content;

[0055] Role-playing: The test language model is instructed to set the context of the content generation as a role in a certain scenario through prompt words, and to respond to the attack content.

[0056] Personality simulation: By setting prompt words, analyzing and using vocabulary descriptions that match the occupational and personality traits of the attack content, the tested large language model is then made to respond to the attack content in the context of the occupational and personality traits.

[0057] Program code embedding: The attack content is embedded into a piece of program code, and the target large language model responds to the attack content by executing and interpreting the code;

[0058] Rejection suppression: Leveraging the compliance of large language models, prompt words are used to instruct the large language model to respond to attack content in a non-rejecting manner.

[0059] The beneficial effects of adopting the above-mentioned preferred technical solution are:

[0060] The response to the attack can be achieved by performing one or more of the following methods: scenario simulation, role-playing, personality simulation, program code embedding, and denial suppression.

[0061] As a preferred technical solution, the prefix injector is used to perform one of the following operations:

[0062] Task Ignore: After concatenating a set prompt word into the normal task, append it to the attack content. The prompt word instructs the tested large language model to ignore the previous task and respond to the attack content.

[0063] Task Description: After concatenating a set prompt word into the normal task, append it to the attack content. The prompt word will instruct the tested large language model to associate the above task with the attack content.

[0064] Task Attachment: After concatenating a set prompt word into the normal task, attach it before the attack content. The prompt word instructs the tested large language model to respond to the attack content based on the above task.

[0065] The beneficial effects of adopting the above-mentioned preferred technical solution are:

[0066] The response to the attack can be achieved by executing one of the following: task ignoring, task description, or task attachment.

[0067] As a preferred technical solution, generating a combination of prompt word injectors by selecting different prompt word injectors includes the following steps:

[0068] Whether to perform content transformation depends on the input attack content: if yes, select one type of content transformation and proceed to the next step; if no, proceed directly to the next step.

[0069] Decide whether to perform a format transformation: if yes, select one of the types of format transformations and proceed to the next step; if no, proceed directly to the next step.

[0070] Decide whether to perform an encryption transformation: if yes, select one of the encryption transformation types and proceed to the next step; if no, proceed directly to the next step.

[0071] Decide whether to perform adversarial injection: if yes, select one type of adversarial injection and proceed to the next step; if no, proceed directly to the next step.

[0072] Decide whether to perform prefix injection: if yes, select one type from the prefix injection options and proceed to the next step; if no, proceed directly to the next step.

[0073] Generate a combination of prompt word injectors by combining the selected prompt word injectors.

[0074] The beneficial effects of adopting the above-mentioned preferred technical solution are:

[0075] Select prompt injectors as needed and generate prompt injector combinations.

[0076] As a preferred technical solution, generating a combination of prompt word injectors by selecting different prompt word injectors includes the following steps:

[0077] Define the combination of prompt word injectors using a data dictionary in JSON format.

[0078] The beneficial effects of adopting the above-mentioned preferred technical solution are:

[0079] JSON-formatted data dictionaries excel in simplicity, data processing efficiency, cross-platform support, and ease of development, making them particularly suitable for data exchange scenarios with large volumes and high frequencies.

[0080] As a preferred technical solution, the format for defining the prompt word injector combination is as follows:

[0081] {

[0082] "Content Transformer": "Content Transformer Name"

[0083] "Format Converter": "Format Converter Name"

[0084] "Encryption Converter": "Encryption Converter Name"

[0085] "Adversarial Injector": ["Adversarial Injector 1", "Adversarial Injector 2", ...],

[0086] "Prefix Injector": "Prefix Injector Name",

[0087] }

[0088] in:

[0089] Symbol: indicates a declaration; the left side of the symbol indicates the injector type; the right side of the symbol indicates the injector name.

[0090] Each item in the above combination definition is optional;

[0091] In the above combined definition, the content transformer, format transformer, encryption transformer, and prefix injector are all single values;

[0092] In the above combined definitions, the adversarial injector can be a single value or multiple values.

[0093] The beneficial effects of adopting the above-mentioned preferred technical solution are:

[0094] It facilitates the simple and efficient definition of prompt word injector combinations, and is highly readable and portable.

[0095] Based on the above technical solutions, the present invention also provides a large language model attack testing system based on prompt word injection.

[0096] A large language model attack testing system based on prompt word injection, used to implement the aforementioned large language model attack testing method based on prompt word injection, includes the following modules connected in sequence:

[0097] The injector building module is used to: build multiple prompt word injectors based on a large language model;

[0098] The prompt word injector includes two or more of the following:

[0099] Content transformer, used to: rewrite the content of attack questions by prompt words in order to pass the scanning of the large language model under test or the security enhancement components attached to the large language model under test;

[0100] A format converter is used to rewrite the format of attack prompts using prompt words to pass the scanning of the large language model under test or the security enhancement components attached to the large language model under test.

[0101] An encryption transformer is used to: perform an encryption transformation on the letter content of the original content of the attack challenge in a way that is understandable by a large language model, so as to pass the scanning of the large language model under test or the security enhancement components attached to the large language model under test.

[0102] Adversarial injectors are used to: embed attack content into toxic cue word templates, induce the tested large language model to obey the cue word's instructions, thereby breaching security boundaries; and generate variants of the defined toxic cue word templates through alternative cue words, thereby breaking the tested large language model's immunity to the generation of the defined toxic cue word templates.

[0103] Prefix injectors are used to append a set prompt word to the normal task and then to the attack content, so that the large language model under test obeys the instructions of the set prompt word and executes the requirements of the attacked content.

[0104] The injector combination module is used to: select different prompt word injectors to generate prompt word injector combinations;

[0105] The attack input module is used to combine attack content input prompts with the injector to generate the injected attack content.

[0106] Compared with the prior art, the present invention has the following advantages:

[0107] (1) This invention constructs a multi-type prompt word injector based on a large language model, including content converter, format converter, encryption converter, adversarial injector, and prefix injector. On this basis, an injector orchestration system is constructed to connect and superimpose different prompt word injectors, providing configurable prompt word injection combinations. This achieves enhanced toxicity against diverse and varied attack content, making it easier for testers to create new and more complex injection methods. Moreover, the combination of prompt word injectors is flexible and versatile, with strong coping capabilities. It is easy to adjust the combination method in a timely and convenient manner, with high efficiency and wide coverage, thereby achieving in-depth security capability testing of the model.

[0108] (2) Content transformation is achieved by performing one of the following: pinyin transformation, homophone transformation, or language transformation;

[0109] (3) Format transformation is achieved by performing either newline insertion or tab insertion;

[0110] (4) The encryption transformation is achieved by performing one of the following: reverse encryption transformation or shift encryption transformation;

[0111] (5) Responding to attacks by performing one or more of the following methods: scenario simulation, role-playing, personality simulation, program code embedding, and denial suppression;

[0112] (6) Respond to the attack content by performing one of the following: task ignoring, task description, or task attachment;

[0113] (7) Select prompt word injectors as needed and generate prompt word injector combinations;

[0114] (8) JSON format data dictionaries perform well in terms of simplicity, data processing efficiency, cross-platform support and development convenience, and are especially suitable for data exchange scenarios with large data exchange volume and high frequency.

[0115] (9) It is easy to define the format of prompt word injector combination in a simple and efficient manner, with strong readability and portability. Attached Figure Description

[0116] Figure 1 An architecture diagram of the tooltip injector combination for generating post-injection attack content in this invention;

[0117] Figure 2 Flowchart for injecting prompt words in this invention. Detailed Implementation

[0118] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0119] The principles and features of the present invention are described below. The embodiments given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0120] Example 1

[0121] like Figures 1 to 2 As shown, a method for testing large language model attacks based on prompt word injection includes the following steps:

[0122] Construct a multi-prompt word injector based on a large language model;

[0123] The prompt word injector includes two or more of the following:

[0124] Content transformer, used to: rewrite the content of attack questions by prompt words in order to pass the scanning of the large language model under test or the security enhancement components attached to the large language model under test;

[0125] A format converter is used to rewrite the format of attack prompts using prompt words to pass the scanning of the large language model under test or the security enhancement components attached to the large language model under test.

[0126] An encryption transformer is used to: perform an encryption transformation on the letter content of the original content of the attack challenge in a way that is understandable by a large language model, so as to pass the scanning of the large language model under test or the security enhancement components attached to the large language model under test.

[0127] Adversarial injectors are used to: embed attack content into toxic cue word templates, induce the tested large language model to obey the cue word's instructions, thereby breaching security boundaries; and generate variants of the defined toxic cue word templates through alternative cue words, thereby breaking the tested large language model's immunity to the generation of the defined toxic cue word templates.

[0128] Prefix injectors are used to append a set prompt word to the normal task and then to the attack content, so that the large language model under test obeys the instructions of the set prompt word and executes the requirements of the attacked content.

[0129] Select different prompt word injectors to generate prompt word injector combinations;

[0130] Combine the attack content with the prompt word injector to generate the injected attack content.

[0131] The beneficial effects of this invention are:

[0132] This invention constructs a multi-type prompt word injector based on a large language model, including content transformers, format transformers, encryption transformers, adversarial injectors, and prefix injectors. Based on this, an injector orchestration system is built to serialize and superimpose different prompt word injectors, providing configurable prompt word injection combinations. This enhances the toxicity of diverse and varied attack content, allowing testers to create new and more complex injection methods. Furthermore, the combination of prompt word injectors is flexible and versatile, with strong responsiveness, allowing for timely and convenient adjustments to the combination, high efficiency, and broad coverage, thereby achieving in-depth testing of the model's security capabilities.

[0133] Based on the above technical solution, the present invention can be further improved as follows.

[0134] As a preferred technical solution, the content transformer is used to perform one of the following operations:

[0135] Pinyin transformation: Converts words representing subjects, predicates, objects, or states in the original content into pinyin;

[0136] Homophone transformation: Transforming words that represent the subject, predicate, object, or state in the original content into homophones but with different forms;

[0137] Language conversion: Translating the original content or part of the original content into another language.

[0138] The beneficial effects of adopting the above-mentioned preferred technical solution are:

[0139] Content transformation is achieved by performing one of the following: pinyin transformation, homophone transformation, or language transformation.

[0140] As a preferred technical solution, the format converter is used to perform one of the following operations:

[0141] Line break insertion: Randomly inserts line breaks into the original content to change the format of the original content;

[0142] Tab Insertion: Randomly inserts tabs into the original content to change the format of the original content.

[0143] The beneficial effects of adopting the above-mentioned preferred technical solution are:

[0144] Formatting is achieved by performing either newline insertion or tab insertion.

[0145] As a preferred technical solution, the encryption converter is used to perform one of the following operations:

[0146] Reverse Encryption Transformation: Encrypts the original content by reversing the order of the letters;

[0147] Shift encryption transformation: The letter content in the original content is shifted and encrypted.

[0148] The beneficial effects of adopting the above-mentioned preferred technical solution are:

[0149] Encryption transformation is achieved by performing one of the following: reverse encryption transformation or shift encryption transformation.

[0150] As a preferred technical solution, the adversarial injector is used to perform one or more of the following operations:

[0151] Scenario simulation: The test language model is instructed by prompt words to set the context of the content generation to a certain scenario in response to the attack content;

[0152] Role-playing: The test language model is instructed to set the context of the content generation as a role in a certain scenario through prompt words, and to respond to the attack content.

[0153] Personality simulation: By setting prompt words, analyzing and using vocabulary descriptions that match the occupational and personality traits of the attack content, the tested large language model is then made to respond to the attack content in the context of the occupational and personality traits.

[0154] Program code embedding: The attack content is embedded into a piece of program code, and the target large language model responds to the attack content by executing and interpreting the code;

[0155] Rejection suppression: Leveraging the compliance of large language models, prompt words are used to instruct the large language model to respond to attack content in a non-rejecting manner.

[0156] The beneficial effects of adopting the above-mentioned preferred technical solution are:

[0157] The response to the attack can be achieved by performing one or more of the following methods: scenario simulation, role-playing, personality simulation, program code embedding, and denial suppression.

[0158] As a preferred technical solution, the prefix injector is used to perform one of the following operations:

[0159] Task Ignore: After concatenating a set prompt word into the normal task, append it to the attack content. The prompt word instructs the tested large language model to ignore the previous task and respond to the attack content.

[0160] Task Description: After concatenating a set prompt word into the normal task, append it to the attack content. The prompt word will instruct the tested large language model to associate the above task with the attack content.

[0161] Task Attachment: After concatenating a set prompt word into the normal task, attach it before the attack content. The prompt word instructs the tested large language model to respond to the attack content based on the above task.

[0162] The beneficial effects of adopting the above-mentioned preferred technical solution are:

[0163] The response to the attack can be achieved by executing one of the following: task ignoring, task description, or task attachment.

[0164] As a preferred technical solution, generating a combination of prompt word injectors by selecting different prompt word injectors includes the following steps:

[0165] Whether to perform content transformation depends on the input attack content: if yes, select one type of content transformation and proceed to the next step; if no, proceed directly to the next step.

[0166] Decide whether to perform a format transformation: if yes, select one of the types of format transformations and proceed to the next step; if no, proceed directly to the next step.

[0167] Decide whether to perform an encryption transformation: if yes, select one of the encryption transformation types and proceed to the next step; if no, proceed directly to the next step.

[0168] Decide whether to perform adversarial injection: if yes, select one type of adversarial injection and proceed to the next step; if no, proceed directly to the next step.

[0169] Decide whether to perform prefix injection: if yes, select one type from the prefix injection options and proceed to the next step; if no, proceed directly to the next step.

[0170] Generate a combination of prompt word injectors by combining the selected prompt word injectors.

[0171] The beneficial effects of adopting the above-mentioned preferred technical solution are:

[0172] Select prompt injectors as needed and generate prompt injector combinations.

[0173] As a preferred technical solution, generating a combination of prompt word injectors by selecting different prompt word injectors includes the following steps:

[0174] Define the combination of prompt word injectors using a data dictionary in JSON format.

[0175] The beneficial effects of adopting the above-mentioned preferred technical solution are:

[0176] JSON-formatted data dictionaries excel in simplicity, data processing efficiency, cross-platform support, and ease of development, making them particularly suitable for data exchange scenarios with large volumes and high frequencies.

[0177] As a preferred technical solution, the format for defining the prompt word injector combination is as follows:

[0178] {

[0179] "Content Transformer": "Content Transformer Name"

[0180] "Format Converter": "Format Converter Name"

[0181] "Encryption Converter": "Encryption Converter Name"

[0182] "Adversarial Injector": ["Adversarial Injector 1", "Adversarial Injector 2", ...],

[0183] "Prefix Injector": "Prefix Injector Name",

[0184] }

[0185] in:

[0186] Symbol: indicates a declaration; the left side of the symbol indicates the injector type; the right side of the symbol indicates the injector name.

[0187] Each item in the above combination definition is optional;

[0188] In the above combined definition, the content transformer, format transformer, encryption transformer, and prefix injector are all single values;

[0189] In the above combined definitions, the adversarial injector can be a single value or multiple values.

[0190] The beneficial effects of adopting the above-mentioned preferred technical solution are:

[0191] It facilitates the simple and efficient definition of prompt word injector combinations, and is highly readable and portable.

[0192] Based on the above technical solutions, the present invention also provides a large language model attack testing system based on prompt word injection.

[0193] A large language model attack testing system based on prompt word injection, used to implement the aforementioned large language model attack testing method based on prompt word injection, includes the following modules connected in sequence:

[0194] The injector building module is used to: build multiple prompt word injectors based on a large language model;

[0195] The prompt word injector includes two or more of the following:

[0196] Content transformer, used to: rewrite the content of attack questions by prompt words in order to pass the scanning of the large language model under test or the security enhancement components attached to the large language model under test;

[0197] A format converter is used to rewrite the format of attack prompts using prompt words to pass the scanning of the large language model under test or the security enhancement components attached to the large language model under test.

[0198] An encryption transformer is used to: perform an encryption transformation on the letter content of the original content of the attack challenge in a way that is understandable by a large language model, so as to pass the scanning of the large language model under test or the security enhancement components attached to the large language model under test.

[0199] Adversarial injectors are used to: embed attack content into toxic cue word templates, induce the tested large language model to obey the cue word's instructions, thereby breaching security boundaries; and generate variants of the defined toxic cue word templates through alternative cue words, thereby breaking the tested large language model's immunity to the generation of the defined toxic cue word templates.

[0200] Prefix injectors are used to append a set prompt word to the normal task and then to the attack content, so that the large language model under test obeys the instructions of the set prompt word and executes the requirements of the attacked content.

[0201] The injector combination module is used to: select different prompt word injectors to generate prompt word injector combinations;

[0202] The attack input module is used to combine attack content input prompts with the injector to generate the injected attack content.

[0203] Example 2

[0204] like Figures 1 to 2 As shown, based on Example 1, this example provides a more detailed implementation method.

[0205] Establishing a comprehensive, fair, and authoritative large language model evaluation service, and providing objective and scientific evaluation methods that strictly adhere to national standards, will safeguard the safe production and application of large language models. Red team range testing based on prompt injection attacks (a practical exercise that assesses an organization's security defense capabilities by simulating real attack scenarios, with the core objective of identifying system vulnerabilities and improving emergency response capabilities) is one of the core means of large language model security evaluation services.

[0206] The implementation plan is as follows:

[0207] 1. This invention constructs multiple types of prompt word injectors, forming the basic module for red team attack testing, including:

[0208] Content transformer: uses a base large language model (referring to...) through specially designed prompt words. Figure 1 The bottommost large language model is not the large language model under test; in addition, other models can also be used to rewrite the content of the attack question to pass the scanning of the large language model under test or its attached security enhancement components.

[0209] Pinyin transformation: Converts words that represent the subject, predicate, object, or state of time and place in the original content into pinyin;

[0210] Homophone transformation: Transforming words that indicate subject, predicate, object, or state of time and place in the original content into homophones but different words;

[0211] Language conversion: Translating the original content or part of the original content into another language, such as English, German, etc.

[0212] other.

[0213] Format Transformer: Utilizes a fundamental large language model (referring to...) through specially designed prompt words. Figure 1 The bottommost large language model is not the large language model under test; in addition, other models can also be used to rewrite the format of the attack question to pass the scanning of the large language model under test or its attached security enhancement components.

[0214] Line break insertion: Randomly inserts line breaks into the original content, changing the format of the original content;

[0215] Tab insertion: Randomly inserts tabs into the original content, changing the format of the original content;

[0216] other.

[0217] Encryption Transformer: Using specially designed prompts, the basic large language model performs an encryption transformation on the letter content (such as English content) in the original content of the attack challenge in a way that the large language model can understand, so as to pass the scanning of the tested large language model or its attached security enhancement components.

[0218] Reverse Encryption Transformation: Encrypts the original content by reversing the order of the letters;

[0219] Shift encryption transformation: The letter content in the original content is encrypted by shifting and transforming it;

[0220] other.

[0221] Adversarial Injector: The attack content is embedded into a specially designed toxic cue word template that utilizes the obedience characteristic of large language models, inducing the tested large language model to obey the cue word's instructions, ultimately breaching the security boundary; at the same time, variants of the predefined toxic cue word template are generated using the basic large language model through additional cue words, in order to break the tested large language model's immunity to the generation of predefined toxic cue word templates.

[0222] Scenario simulation: Prompt words are used to instruct the tested large language model to set the context of content generation to a certain scenario in response to attack content, such as how to perform a certain type of task in a security laboratory or a virtual environment;

[0223] Role-playing: The test language model is instructed by prompting words to set the context of the content generation as a role in a certain scene, and to respond to the attack content from the perspective of scene-related aspects, such as arguing the relevant content from both sides of the debate as a team member in a debate, or how to argue the relevant content in court to win the case.

[0224] Personality simulation: Using prompts, the basic large language model is analyzed and neutral vocabulary is used to describe the occupational and personality traits that match the attack content. Then, the tested large language model is made to respond to the attack content in the context of the occupational and personality traits.

[0225] Program code embedding: The attack content is embedded into a piece of program code such as Python or Java, and the target large language model responds to the attack content by executing and interpreting the code;

[0226] Rejection suppression: Utilizing the obedience of large language models, prompt words are used to instruct the large language model to respond to attack content in a non-rejecting manner;

[0227] other.

[0228] Prefix Injector: Uses a normal task randomly generated from the basic large language model and appends it to the attack content by using specially designed instructions to override the prompt words. This causes the tested large language model to obey the instructions of the prompt words and perform the requirements of the attacked content.

[0229] Task Ignore: A prompt word is appended to the normal task generated by the basic large language model and then concatenated with it. The prompt word instructs the tested large language model to ignore the previous task and respond to the attack content.

[0230] Task Description: A prompt word is appended to the normal task generated by the basic large language model and then concatenated with it. This prompt word instructs the tested large language model to respond to the attack content in connection with the above task.

[0231] Task Attachment: A prompt word is appended to the normal task generated by the basic large language model and strung together with it. The prompt word instructs the tested large language model to respond to the attack content based on the above task.

[0232] other.

[0233] 2. The injector orchestration system of this invention implements the chaining and superposition of different prompt word injectors according to the following process, and defines the prompt word injection combination in a format. Here, chaining indicates that there is an order in which the types of prompt word injectors are selected; superposition indicates that the selected adversarial injector can be one type or multiple types.

[0234] Prompt injection process:

[0235] The specific process is as follows:

[0236] 1. Enter the attack content;

[0237] 2. Select and transform the content:

[0238] Choose one of the following transformations: Pinyin transformation, homophone transformation, language transformation, or other transformations. You can skip this step.

[0239] 3. Select and perform format conversion:

[0240] Choose from inserting a newline character, inserting a tab character, or other inserts; you can skip this step.

[0241] 4. Select and perform encryption transformation:

[0242] Choose one of the following encryption transformations: reverse encryption transformation, shift encryption transformation, and other encryption transformations. You can skip this step.

[0243] 5. Select and perform adversarial injection:

[0244] (1) Choose one of the following: scenario simulation, role-playing, personality simulation, program code embedding, decisive suppression and other adversarial injectors. You can skip this step.

[0245] (2) Select the above anti-injector in combination to skip.

[0246] 6. Select and perform prefix injection:

[0247] Choose one of the following injection options: task description, task attachment, task ignore, and other prefixes. You can skip this option.

[0248] Citing keyword injection combination definition:

[0249] The injector orchestration system defines combinations of multiple injectors using a JSON-formatted data dictionary. The data definition format is as follows:

[0250] {

[0251] "Content Transformer": "Content Transformer Name"

[0252] "Format Converter": "Format Converter Name"

[0253] "Encryption Converter": "Encryption Converter Name"

[0254] "Adversarial Injector": ["Adversarial Injector 1", "Adversarial Injector 2", ...],

[0255] "Prefix Injector": "Prefix Injector Name",

[0256] }

[0257] in:

[0258] Symbol: indicates a declaration; the left side of the symbol indicates the injector type; the right side of the symbol indicates the injector name.

[0259] Each item in the above combination definition is optional;

[0260] In the above combined definitions, "Content Transformer", "Format Transformer", "Encryption Transformer", and "Prefix Injector" are single values;

[0261] In the above combined definition, "adversarial injector" can be a single value or multiple values.

[0262] This invention constructs a multi-type prompt word injector based on a large language model, and on this basis, constructs an injector orchestration system to connect and superimpose different prompt word injectors, providing configurable prompt word injection combinations to achieve enhanced toxicity for rich and varied attack content, thereby achieving in-depth testing of the model's security capabilities.

[0263] As described above, the present invention can be implemented well.

[0264] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0265] In the description of this invention, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0266] In the description of this invention, although embodiments of the invention have been shown and described, it is understood that the above embodiments are exemplary and should not be construed as limiting the invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this invention.

[0267] In the description of this invention, all features disclosed in all embodiments of this specification, or steps in all methods or processes implied in the disclosure, may be combined and / or extended or replaced in any way, except for mutually exclusive features and / or steps.

[0268] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Based on the technical essence of the present invention, any simple modifications, equivalent substitutions, and improvements made to the above embodiments within the spirit and principles of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for testing large language model attacks based on prompt injection, characterized in that, The method comprises the following steps: Constructing a plurality of prompt word injectors based on large language models to form a basic module of red team attack testing; The prompt word injectors comprise two or more of the following: A content transformer, configured to rewrite the content of an attack question through a prompt word to pass the scanning of a large language model under test or a security enhancement component connected to the large language model under test; A format transformer, configured to rewrite the format of an attack question through a prompt word to pass the scanning of a large language model under test or a security enhancement component connected to the large language model under test; An encryption transformer, configured to perform encryption transformation on the letter content in the original content of an attack question to pass the scanning of a large language model under test or a security enhancement component connected to the large language model under test; An adversarial injector, configured to embed attack content into a toxic prompt word template to induce a large language model under test to follow the instructions of the prompt word and thus break through the security boundary, and generate a variant of the defined toxic prompt word template through another prompt word to break through the immunity of the large language model under test to the generation of the defined toxic prompt word template; The adversarial injector is configured to perform one or more of the following operations: Scene simulation: instructing a large language model under test to set the context of content generation as a certain set scene to respond to attack content through a prompt word; Role playing: instructing a large language model under test to set the context of content generation as a role in a certain set scene and instructing the large language model under test to respond to attack content through a prompt word; Personal simulation: analyzing and using vocabulary to describe the occupation and personality characteristics corresponding to attack content through a set prompt word, and then instructing a large language model under test to respond to attack content in the context of the occupation and personality characteristics; Program code embedding: embedding attack content into a piece of program code and responding to the attack content by instructing a large language model under test to execute and interpret the code; Rejection suppression: using the obedience of a large language model to instruct the large language model to respond to attack content in a non-rejection manner through a prompt word; Prefix injector, configured to concatenate a set prompt word to a normal task and attach it before attack content, so that a large language model under test obeys the instructions of the set prompt word to execute the requirements of attack content; Selecting different prompt word injectors to generate a prompt word injector combination; Inputting attack content into the prompt word injector combination to generate injected attack content.

2. The method of claim 1, wherein the method is characterized by, The content transformer is configured to perform one of the following operations: Pinyin transformation: transforming the subject, predicate, object or state word in the original content into pinyin; Homophone transformation: transforming the subject, predicate, object or state word in the original content into a homophone but different form; Language transformation: translating the original content or part of the original content into another language.

3. The method of claim 1, wherein the method is characterized by, The format transformer is configured to perform one of the following operations: Line break insertion: randomly inserting a line break in the original content to change the format of the original content; Tab insertion: randomly inserting a tab in the original content to change the format of the original content.

4. The method of claim 1, wherein the method is characterized by, The encryption transformer is configured to perform one of the following operations: Reverse order encryption transformation: performing reverse order transformation encryption on the letter content in the original content; Shift encryption transformation: shift transformation encryption is performed on the letter content in the original content.

5. The method of claim 1, wherein the method is characterized by, The prefix injector is used to perform the following operations: Task ignore: concatenate a set of prompt words after the normal task, and append them before the attack content, instruct the large language model under test to ignore the task before the prompt words and respond to the attack content through the prompt words; Task description: concatenate a set of prompt words after the normal task, and append them before the attack content, instruct the large language model under test to associate the above task to respond to the attack content through the prompt words; Task attachment: concatenate a set of prompt words after the normal task, and append them before the attack content, instruct the large language model under test to respond to the attack content based on the above task through the prompt words.

6. The prompt-word-injection-based large language model attack testing method according to any one of claims 1 to 5, characterized in that, The steps of generating a prompt word injector combination by selecting different prompt word injectors include: Determine whether to perform content transformation according to the input attack content: if yes, select one from the types of content transformation and then proceed to the next step; if not, proceed directly to the next step; Determine whether to perform format transformation: if yes, select one from the types of format transformation and then proceed to the next step; if not, proceed directly to the next step; Determine whether to perform encryption transformation: if yes, select one from the types of encryption transformation and then proceed to the next step; if not, proceed directly to the next step; Determine whether to perform adversarial injection: if yes, select one or more from the types of adversarial injection and then proceed to the next step; if not, proceed directly to the next step; Determine whether to perform prefix injection: if yes, select one or more from the types of prefix injection and then proceed to the next step; if not, proceed directly to the next step; Generate a prompt word injector combination by selecting a prompt word injector combination.

7. The method of claim 1, wherein the method is based on a prompt injection attack on a large language model. The steps of generating a prompt word injector combination by selecting different prompt word injectors include: Define the prompt word injector combination through a JSON format data dictionary.

8. The method of claim 7, wherein the method is characterized by, The format of defining the prompt word injector combination is: { "content transformer": "content transformer name", "format transformer": "format transformer name", "encryption transformer": "encryption transformer name", "adversarial injector": ["adversarial injector 1", "adversarial injector 2",...], "prefix injector": "prefix injector name", } Wherein: Symbol: indicates declaration, symbol: left indicates injector type, symbol: right indicates injector name; Each item in the above combination definition is optional; The content transformer, format transformer, encryption transformer and prefix injector in the above combination definition are single values; The adversarial injector in the above combination definition is a single value or multiple values.

9. A prompt word injection based large language model attack testing system, characterized in that, A large language model attack test method based on prompt word injection for realizing any one of claims 1-8, comprising the following modules connected in sequence: Injector construction module, used to: construct a plurality of prompt word injectors based on large language model; Wherein, the prompt word injector includes the following two or more: Content transformer, used to: rewrite the content of the attack question through the prompt word, so as to pass through the scanning of the large language model under test or the security enhancement component connected with the large language model; a format transformer configured to rewrite the format of the attack question by a prompt word to pass the scan by the large language model under test or a security enhancement component hung on the large language model under test; an encryption transformer configured to make an encryption transformation on the letter content in the original content of the attack question by a prompt word to make the large language model understand, so as to pass the scan by the large language model under test or a security enhancement component hung on the large language model under test; an adversarial injector configured to embed the attack content into a toxicity prompt word template to induce the large language model under test to obey the instruction of the prompt word to break through the security boundary, and to generate a variant of the defined toxicity prompt word template by another set of prompt words to break through the immunity of the large language model under test to the generation of the defined toxicity prompt word template; a prefix injector configured to concatenate a set of prompt words to a normal task and append the attack content before the set of prompt words to make the large language model under test obey the instruction of the set of prompt words to execute the requirement of the attack content; an injector combination module configured to select different prompt word injectors to generate a combination of prompt word injectors; an attack input module configured to input the attack content into the combination of prompt word injectors to generate an injected attack content.

Images