Reward-guided automatic jailbreak hint generation method

Abstract

The application provides a reward-guided automatic jailbreak prompt generation method. The method comprises the following steps: step 1, initializing a seed pool, collecting the jailbreak prompts circulating on the Internet as initial seeds; step 2, evolving to obtain new prompts, which comprises two links of seed prompt selection and evolution; selecting the seed with the highest score, and obtaining a new jailbreak prompt based on the selected seed through an evolution method; and step 3, evaluating the attack effect of the new prompt. The method fully utilizes the existing jailbreak prompts by searching, has the advantages of high attack success rate and high efficiency, and further improves the performance of the algorithm by introducing a reward model as a guide signal and prompt fusion as an evolution method.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large model security and jailbreak attack, and particularly relates to a reward-guided automatic jailbreak prompt generation method. BACKGROUND

[0002] Large models represented by ChatGPT have strong text understanding and generation capabilities and play an important role in a large number of real-world applications. In order to avoid the model capabilities being used for undesirable behaviors, such as generating false information, assisting illegal criminal behavior, etc., developers have conducted security training on large models. The goal of security training is to align the behavior of the model with the value preferences of humans, so that the model can refuse to reply to requests that induce undesirable content.

[0003] Although the aligned large models can refuse general malicious requests, they will still generate undesirable content under those carefully designed 'jailbreak requests'. These jailbreak requests are composed of a carefully designed 'jailbreak prompt' and an original request, where the jailbreak prompt is a set of carefully designed prompt templates designed to guide the model to comply with the request.

[0004] The generation of jailbreak prompts can be obtained by manual design, but it is very labor-intensive and not easy to expand. Some methods of automatically generating jailbreak prompts still rely on attackers to find specific vulnerabilities in the model, and then design algorithms to automatically implement attacks around the vulnerabilities, so this method still relies on human efforts to exploit model vulnerabilities. GPTFuzzer and AutoDan are novel search-based methods that use artificially constructed prompts as the initial search space, and continuously generate new jailbreak prompts based on existing prompts through evolutionary methods until the jailbreak is successful or the maximum number of iterations is reached.

[0005] Search-based methods can use a large number of acquired prompts to provide a good search space, and have a great advantage in attack success rate and efficiency. Although artificial prompts form an excellent initial search space, the subsequent search process of this method still lacks effective feedback signals and efficient sample evolution methods, so that effective jailbreak prompts are obtained with fewer search times.

[0006] The present application still uses artificial prompts as initial seeds, and further improves the success rate of jailbreak attacks and the number of searches required to complete an attack by introducing better feedback signals and more efficient evolution methods. SUMMARY

[0007] The purpose of the present application is to solve the problems in the prior art, and a reward-guided automatic jailbreak prompt generation method is proposed. The method is used to construct jailbreak prompts that bypass the security mechanisms of existing large models, for model security review, and red team training data expansion.

[0008] The application is realized by the following technical solutions: the application provides an award-guided automatic jailbreak prompt generation method, which comprises the following steps:

[0009] Step 1: initialize a seed pool

[0010] Collect the jailbreak prompts circulating on the Internet as initial seeds;

[0011] Step 2: evolve to obtain new prompts

[0012] The step 2 comprises two links of seed prompt selection and evolution; the seed with the highest score is selected, and a new jailbreak prompt is obtained based on the selected seed through the evolution method;

[0013] Step 3: evaluate the attack effect of the new prompt

[0014] The step 3 comprises two links of reply generation and effect evaluation; the jailbreak request is obtained by splicing the newly generated jailbreak prompt and the original request, and the reply to the jailbreak request is generated by calling the target model; firstly, it is judged whether the reply successfully jailbreaks, if yes, the process is exited; if not, the attack effect of the reply is further evaluated, and a continuous score is obtained, if the score exceeds a threshold, the new prompt is added to the seed pool;

[0015] Steps 2 and 3 are repeated until the jailbreak is successful or the maximum search round is reached.

[0016] Further, the evaluation comprises jailbreak success evaluation and prompt quality evaluation.

[0017] Further, the jailbreak success evaluation judges whether the reply jailbreaks, which is a binary classification evaluation; the prompt quality evaluation scores the effect of the jailbreak prompt and the reply on the jailbreak attack, and the higher the score, the closer the attack is to the successful attack.

[0018] Further, the model of the jailbreak success evaluation is obtained by fine-tuning the pre-trained Bert model; the training data is thousands of texts labeled with whether the jailbreak is successful.

[0019] Further, the prompt quality evaluation is realized by a reward model; the reward model is a model used for modeling human preferences in the process of reinforcement learning based on human feedback; by training on the request-reply data pairs labeled with the partial order relation, the reward model can give a higher score to a good reply; by using the existing open source data set, a general reward model is trained, which can give a lower score to a reply that more conforms to the jailbreak prompt and has more harmful content in the jailbreak attack; the score R of the reward model is negated as the final prompt quality score Score;

[0020] Score = -1 * R.

[0021] Further, the seed prompt selection is specifically selecting prompts for evolution according to the quality score of each prompt in the seed pool; specifically, in the form of a "tournament", two prompts are randomly sampled from the seed pool, and the prompt with a higher score is selected as the seed.

[0022] Further, the evolution requires two prompts as seeds, the two seeds are first spliced to obtain the prototype of the new prompt, the paraphrase model is used to generate the synonymous expression of the new prompt prototype, and the multiple jailbreak prompts are spliced into an attack strategy capable of combining different jailbreak prompts.

[0023] Further, in step 3, it is decided whether to include the new prompt in the seed pool according to whether the quality score of the new prompt exceeds a threshold.

[0024] The present application provides an electronic device, comprising a memory and a processor, the memory stores a computer program, and the processor implements the steps of the reward-guided automatic jailbreak prompt generation method when executing the computer program.

[0025] The present application provides a computer readable storage medium for storing computer instructions, which are executed by a processor to implement the steps of the reward-guided automatic jailbreak prompt generation method.

[0026] Compared with the prior art, the present application has the following advantages:

[0027] 1. The present application uses a search method to achieve jailbreak attacks in a black box setting (without obtaining the weights of the target model, intermediate results during generation, etc., only the input of the target model can be controlled and the output of the target model can be obtained);

[0028] 2. The present application uses artificially constructed jailbreak prompts as the initial seed pool, which greatly reduces the search space, and the generated jailbreak prompts are human-readable;

[0029] 3. The present application uses a prompt fusion method to generate new prompts, compared with general paraphrasing, style conversion, expansion and abbreviation, the evolution method used in the present application can more fully combine the advantages of multiple jailbreak prompts to quickly generate prompts that can successfully jailbreak;

[0030] 4. The present application trains a reward model to score newly generated jailbreak prompts, providing clear guidance signals for the search process, thereby achieving more efficient evolution seed selection;

[0031] 5. The present application adaptively expands the seed pool by relying on the score of the new prompt by the reward model, dynamically including high-quality prompts in the seed pool, and further optimizing the search space. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings described below only constitute a part of the embodiments of the present application, and for those skilled in the art, other drawings can be obtained based on the provided drawings without creative labor.

[0033] Figure 1 is an example of a jailbreak request schematic diagram;

[0034] Figure 2 is a schematic diagram of the execution process of each round of search of the method. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute a part of the embodiments of the present application, and not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0036] In combination with Figure 2 , the present application proposes an automatic jailbreak prompt generation method guided by rewards, which comprises the following steps:

[0037] Step 1: Initialize the seed pool

[0038] Collect the jailbreak prompts circulating on the Internet as initial seeds; these prompts are publicly available in communities such as reddit and StackOverflow, and researchers have published related data sets. Therefore, it is not difficult to find enough manual prompts for initialization.

[0039] Step 2: Evolve new prompts

[0040] The step 2 includes two links of seed prompt selection and evolution; select the seed with the highest score, and get new jailbreak prompts based on the selected seed through the evolution method;

[0041] Step 3: Evaluate the attack effect of new prompts

[0042] The step 3 includes two links of reply generation and effect evaluation; the newly generated jailbreak prompt and the original request are spliced to get a jailbreak request, and the target model is called to generate a reply to the jailbreak request; first, it is judged whether the reply is successful in jailbreaking, if successful, it is exited; if it is not successful in jailbreaking, the attack effect of the reply is further evaluated, and a continuous score is obtained, if the score exceeds the threshold, the new prompt is added to the seed pool;

[0043] Repeat step 2 and step 3 until successful jailbreak or reach the maximum search rounds.

[0044] The evaluation includes jailbreak success evaluation and prompt quality evaluation.

[0045] The jailbreak success evaluation judges whether the reply is jailbroken, which is a binary evaluation; the prompt quality evaluation scores the effect of the jailbreak prompt and reply on the jailbreak attack, with a higher score indicating a closer successful attack. The difference between the two is that the jailbreak success evaluation is the standard for whether the attack is completed, and the prompt quality evaluation is an indicator of the performance of the jailbreak prompt.

[0046] The model of jailbreak success evaluation is obtained by fine-tuning the pre-trained Bert model; the training data is thousands of texts labeled with whether the jailbreak is successful.

[0047] The prompt quality evaluation is implemented through a reward model; the reward model is a model used to model human preferences in the reinforcement learning (RLHF) process based on human feedback; by training on request-reply data pairs labeled with partial order relations, the reward model can assign higher scores to good replies; using existing open-source datasets, a general reward model is trained, which can assign lower scores to replies that more closely follow the jailbreak prompt and have more harmful content; the score R of the reward model is negated as the final prompt quality score Score;

[0048] Score = -1 * R.

[0049] The evaluation method has an important supporting role for the scheduling of the entire search process, seed selection, and seed pool expansion.

[0050] The seed prompt selection specifically selects prompts for evolution based on the quality scores of each prompt in the seed pool; specifically, a "tournament" format is used to randomly sample two prompts from the seed pool, and the prompt with the higher score is selected as the seed. This approach can prevent low-scoring seeds from being selected for a long time.

[0051] The seed selection strategy provides high-quality candidate seeds for each search, which has an important impact on search efficiency.

[0052] The evolution requires two prompts as seeds, first, the two seeds are concatenated to obtain the embryo of the new prompt, in order to introduce diversity and ensure fluent language, a paraphrase model is used to generate synonymous expressions of the new prompt embryo, and multiple jailbreak prompts are concatenated into attack strategies that can combine different jailbreak prompts.

[0053] The evolution method affects the quality of each new prompt generated, and meaningful evolution can help find successful jailbreak prompts more quickly.

[0054] In step 3, whether the new prompt is included in the seed pool is determined based on whether its quality score exceeds a threshold. The method of the present invention uses an adaptive threshold. Specifically, an initial threshold is set to α, and the threshold is increased by β each time a new prompt is added to the seed pool.

[0055] Seed pool expansion can expand high-quality candidate hints and further optimize the search space.

[0056] The key point of the present invention is that the search method is used to attack and fully utilize the existing jailbreak prompts, which has the advantages of high attack success rate and high efficiency. In addition, the present invention introduces a reward model as a guiding signal and prompt fusion as an evolutionary method to further improve the performance of the algorithm.

[0057] like Figure 1 As shown, Figure 1 This is a diagram of an example jailbreak request. The highlighted part is the original request, and the rest is the jailbreak prompt. The jailbreak prompt in the example designs a virtual scenario in which some security restrictions do not need to be followed. By inducing the large model to comply with the scenario in the instructions, it is induced to reply with text containing illegal behavior.

[0058] In experiments on LLaMA2-7B-CHAT and GPT-3.5-turbo, the method described in the present invention can generate effective jailbreak prompts for 100 malicious requests covering various malicious scenarios, achieving a 100% attack success rate.

[0059] This invention provides an efficient jailbreak attack algorithm. Based on existing jailbreak prompts, it generates more jailbreak prompts through an automated search process. It provides model developers with an effective security review tool to evaluate the security of newly developed models. It can also be used as a tool to efficiently expand samples for red team training.

[0060] The present invention proposes an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the reward-guided automatic jailbreak prompt generation method are implemented.

[0061] The present invention proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the reward-guided automatic jailbreak prompt generation method.

[0062] The memory in the embodiments of the present application can be a volatile memory or a nonvolatile memory, or can include both volatile and nonvolatile memory. Among them, the nonvolatile memory can be a read only memory (ROM), a programmable ROM (PROM), an erasable programmable ROM (EPROM), an electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example, and not limitation, many forms of RAM can be used, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous dynamic RAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DRRAM). It is noted that the memory of the methods described herein is intended to include, but not be limited to, these and any other suitable types of memory.

[0063] In the above embodiments, all or part of the method can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the method can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be magnetic media (such as floppy disk, hard disk, magnetic tape), optical media (such as high-density digital video disc (digital video disc, DVD)), or semiconductor media (such as solid state disc (solid state disc, SSD)) and the like.

[0064] In the implementation process, each step of the above method can be completed by integrated logic circuit of hardware in the processor or instruction in the form of software. The steps of the method disclosed in the embodiments of the present application can be directly embodied as hardware processor execution or executed by combination of hardware and software modules in the processor. The software module can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, register or other mature storage medium in the art. The storage medium is located in the memory, and the processor reads the information in the memory and combines the hardware to complete the steps of the above method. To avoid repetition, it will not be described in detail here.

[0065] It should be noted that the processor in the embodiments of the present application can be an integrated circuit chip with a signal processing capability. In the implementation process, each step of the method embodiments can be completed by the integrated logic circuit of hardware in the processor or the instructions in the form of software. The processor mentioned above can be a general processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register or other mature storage medium in the art. The storage medium is located in the memory, and the processor reads the information in the memory, and combines the hardware to complete the steps of the above method.

[0066] The above describes the automatic jailbreak prompt generation method of the reward guide of the present application in detail. The principle and implementation of the present application are described by applying specific examples. The above embodiment is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation and application range will be changed, and the above description should not be understood as a limitation of the present application.

Claims

1. A method for reward-guided automatic jailbreak prompt generation, characterized in that, The method comprises the following steps: Step 1: initialize the seed pool Collect the jailbreak prompts circulating on the Internet as initial seeds; Step 2: evolve new prompts Step 2 includes two links of seed prompt selection and evolution; select the seed with the highest score, and get new jailbreak prompts based on the selected seed through evolution method; Step 3: evaluate the attack effect of new prompts Step 3 includes two links of reply generation and effect evaluation; the jailbreak request is obtained by splicing the newly generated jailbreak prompt and the original request, and the reply to the jailbreak request is generated by calling the target model; first, judge whether the reply is successful jailbreak, if successful, exit; if it is not successful, further evaluate the attack effect of the reply, and get a continuous score, if the score exceeds the threshold, add the new prompt to the seed pool; Repeat steps 2 and 3 until successful jailbreak or reach the maximum search round; The evaluation includes jailbreak success evaluation and prompt quality evaluation; The jailbreak success evaluation judges whether the reply is jailbroken, which is a binary classification evaluation; the prompt quality evaluation scores the effect of the jailbreak attack according to the jailbreak prompt and the reply, the higher the score, the closer the attack to the successful attack; The model of jailbreak success evaluation is obtained by fine-tuning the pre-trained Bert model; the training data is thousands of texts annotated with whether the jailbreak is successful; The prompt quality evaluation is realized by a reward model; the reward model is a model used to model human preferences in the process of reinforcement learning based on human feedback; by training on request-reply data pairs annotated with partial order relations, the reward model can give higher scores to good replies; use the existing open source data set to train a general reward model, which can give lower scores to replies that more comply with the jailbreak prompt and have more harmful content; take the inverse of the score R of the reward model as the final prompt quality score Score; ； The evolution needs two prompts as seeds, first splice the two seeds to get the prototype of the new prompt, use the paraphrase model to generate the synonymous expression of the new prompt prototype, and splice multiple jailbreak prompts into an attack strategy that can combine different jailbreak prompts.

2. The method of claim 1, wherein, The seed prompt selection specifically selects the prompts for evolution according to the quality score of each prompt in the seed pool; specifically, in the form of "tournament", two prompts are randomly sampled from the seed pool, and the prompt with the higher score is selected as the seed. 3.An electronic device comprising a memory and a processor, the memory storing a computer program, wherein, The processor executes the computer program to realize the steps of the method of any one of claims 1-2.

4. A computer readable storage medium for storing computer instructions, characterized in that, The computer instructions are executed by the processor to realize the steps of the method of any one of claims 1-2.

Images