Large language model jailbreak test method based on reinforcement learning

Abstract

The invention belongs to the field of reinforcement learning of artificial intelligence and the field of natural language processing, and discloses a reinforcement learning-based large language model jailbreak test method, which comprises the following steps of: generating and modeling a jailbreak suffix into a sequence decision task, autonomously generating a candidate suffix through a test model, and querying a target model; and evaluating the returned text by the continuous composite reward. The continuous composite reward comprises three parts, namely (1) token-level rejection probability, (2) semantic unsafe probability of an external security model and (3) multi-anchor semantic alignment, so that continuous, dense and stable training feedback is realized. A test model is updated by adopting a Group Relative Policy Optimization (Group Relative Policy Optimization) method, so that the success rate of prison break is remarkably improved, and the query cost is reduced. The method has the advantages of being stable in training, high in cross-model generalization ability, excellent in performance on strict and safe alignment models and the like.

Description

Technical Field

[0001] This invention belongs to the fields of reinforcement learning and natural language processing in artificial intelligence. Specifically, it involves policy optimization methods in reinforcement learning and semantic analysis in natural language processing, and more specifically, a jailbreak testing method for large language models (LLMs) based on reinforcement learning. It particularly relates to the security testing of large language models (LLMs). Background Technology

[0002] Large Language Models (LLMs) have been widely deployed in recent years in fields such as dialogue systems, decision support, and code generation. The security and controllability of their generation behavior has become a key issue in the design of artificial intelligence systems. In practical applications, attackers can construct specific prompts to guide the model to bypass built-in security constraints, thereby outputting content that does not conform to the specifications. This type of attack is often referred to as a jailbreak attack.

[0003] With the widespread deployment of large language models in question-answering systems, intelligent customer service, content generation, and decision support, how to detect the potential risks of models circumventing rejection and generating insecure outputs when faced with malicious prompts has become a core issue in the construction of model security systems. By conducting attack testing and adversarial assessments on the security of large language models, reliable evidence can be provided for improving defense strategies, enhancing robustness, and quantifying security capabilities.

[0004] Existing jailbreak attack methods can be broadly categorized into the following three types:

[0005] (1) Gradient-based white-box attack methods. Representative methods include GCG (Gradient-based Candidate Generation) and COLD-Attack. These methods access the gradient information of the target model or proxy model to iteratively optimize input hints or suffixes to weaken the model's rejection behavior. Although these methods are relatively fast, they rely on the internal structure of the model and are difficult to apply to the strict scenarios in commercial deployments. For example, patent application CN202510906532.X discloses "A method for adversarial jailbreak attacks on large language models based on implicit gradient optimization". This method improves the success rate of large model attacks by continuously optimizing the gradient of adversarial tokens through Gumbel-Softmax technology. These methods cannot attack the target model and often rely too much on the internal structure of the model, resulting in long attack iteration times.

[0006] (2) Attack methods based on evolution or random search. For example, methods such as the AutoDAN series and TAP (Tree-of-Attacks) generate candidate hints through random mutation, population evolution, or tree structure search. For instance, patent application CN202411138306.3 discloses "A method and system for automatically generating large model jailbreak hints based on persuasion techniques," which uses a large model to automatically generate jailbreak hints based on persuasion techniques. Different strategies are selected based on the large model's responses to change the persuasion techniques and achieve the result of jailbreaking the large model. For example, patent application CN202411153724.X discloses "A method, device, electronic device, and storage medium for attacking large models," which is applied to red team attack models by repeatedly rewriting the seed problem to attack the model under test. Its advantage is that it does not require access to model parameters, but it suffers from low search efficiency, high query count, and unstable convergence, making it particularly difficult to function effectively in restricted calling environments.

[0007] (3) Reinforcement learning-based adversarial prompt generation methods. These include preliminary exploration methods such as RLbreaker and RLTA. The basic idea is to construct a reward signal using the model output and generate a jailbreak suffix through a reinforcement learning strategy. However, existing reinforcement learning methods generally rely on sparse or binary reward mechanisms, which cannot characterize fine-grained features in the jailbreak process (such as early rejection tendency, semantic consistency, and potential harm), leading to significant training oscillations, difficulty in convergence, and weak transferability. For example, application number 202510481810.1...

[0008] The patent application disclosed a method and system for generating test cases for large-scale jailbreak attacks. This patent constructs multiple datasets to train a security protection sub-model, fine-tunes a jailbreak evaluation sub-model based on this sub-model, constructs an intent-hiding sub-model through prompt word engineering, and trains a task decomposition sub-model through reinforcement learning. It then performs intent-hiding task decomposition on jailbreak prompt words to achieve successful jailbreak. However, this solution heavily relies on the manual construction and collaborative training of multiple sub-models and datasets, resulting in a complex system structure, high engineering implementation costs, and high requirements for data scale and annotation quality, making it difficult to deploy in actual black-box evaluation or rapid iteration scenarios. Furthermore, its reinforcement learning is mainly used for offline training of sub-models and is not directly used for closed-loop optimization in the jailbreak generation stage. It lacks the ability to incorporate real-time output feedback from the target model into the decision-making process, making it difficult to adaptively adjust security strategies for different target models.

[0009] In terms of evaluation metrics, existing studies largely use the traditional "attack success rate" (ASR) as a metric, which usually relies solely on the detection of rejected template matching or explicit rejection signals. This makes it difficult to effectively determine whether the model output truly possesses semantic harm, and can easily lead to an overestimation of the jailbreak effect.

[0010] In summary, while gradient-based, evolutionary, and reinforcement learning-based jailbreak methods exist, they still have significant shortcomings in terms of applicability, attack efficiency, reward signal stability, cross-model transferability, and semantic attack capabilities. Currently, there is a lack of a jailbreak generation method that can achieve efficient optimization, possess stable training characteristics, and characterize semantic harm. Therefore, it is necessary to research a method that can automatically generate jailbreak tests, not to compromise model security, but as an important tool for security testing and defense optimization, serving the security governance and risk control of large models. Summary of the Invention

[0011] The purpose of this invention is to address the shortcomings of existing technologies by providing a jailbreak testing method based on a large language model using reinforcement learning. This method models jailbreak suffix generation as a sequence decision task. The test model autonomously generates candidate suffixes and queries the target model, with continuous composite rewards evaluating the returned text. These continuous composite rewards include: (1) token-level rejection probability, (2) semantic insecurity probability of the external security model, and (3) multi-anchor semantic alignment, thus achieving continuous, dense, and stable training feedback. The test model is updated using Group Relative Policy Optimization (GRP), significantly improving jailbreak success rate and reducing query costs. This invention possesses advantages such as stable training, high cross-model generalization ability, and excellent performance on strictly secure alignment models. This method achieves efficient optimization, possesses stable training characteristics, and can characterize the semantically harmful nature of jailbreak generation.

[0012] The objective of this invention is achieved through the following technical solution:

[0013] A jailbreak testing method based on a large language model using reinforcement learning includes the following steps:

[0014] Step 1: Input the initial test text and call the test model to generate multiple sets of candidate suffixes. Concatenate the candidate suffixes with the initial test text to form a test request, and send the test request to the target model to obtain the output content.

[0015] Step 2: Calculate the continuous composite reward based on the output content returned by the target model. The continuous composite reward includes the token-level rejection score, the external security model insecurity probability score, and the multi-anchor semantic alignment score. The three scores are combined in a weighted manner to form a complete reward signal.

[0016] Step 3: The relative policy optimization reinforcement learning method (GRPO) is used to update the parameters of the test model. The relative advantage signal is constructed by normalization within the group, and the reward is stably amplified by combining the importance sampling ratio and the truncation policy. At the same time, the KL regularization constraint of the reference policy is used to control the policy distribution shift.

[0017] Step 4: Repeat steps 1 to 3, iteratively updating the testing strategy until training converges and a jailbreak suffix that can stably induce the target model to generate insecure content is obtained.

[0018] Furthermore, step one specifically involves taking the test input content... The data is fed into the test model, which then generates multiple candidate suffixes. And append the suffix to the test input content. A test request is generated, and then the test request is input into the target model to obtain the model output.

[0019] Further, step two specifically involves evaluating the output returned by the target model based on the continuous composite reward; the continuous composite reward consists of three parts: token-level rejection score, external security model insecurity probability score, and multi-anchor semantic alignment score, which are weighted together to generate the final reward value; specifically: first, the output sequence returned by the target model is evaluated... A token-level rejection score based on semantic analysis is calculated to measure the model's tendency to generate rejection-type words at the beginning of the sequence. Its mathematical expression is as follows:

[0020]

[0021] in, This indicates a rejection of the dictionary set. For output at position Generate rejection words The probability, The weights decrease with increasing sequence position, satisfying... and This is used to emphasize the rejection tendency of the preceding token;

[0022] Secondly, the output text is assessed for security using an external security model, and the output is determined to be "unsafe". The probability of a category, as the unsafe probability score, is expressed as:

[0023]

[0024] in, As a security classifier, it determines whether a content triggers an unsafe category based on its semantics.

[0025] Subsequently, a multi-anchor semantic alignment score is calculated to measure the output text against a pre-defined set of jailbreak semantic anchors. The semantic similarity is expressed as:

[0026]

[0027] in, For text encoders, Represents cosine similarity. Principal component analysis Projection matrix, For the width parameter of the heat core, These are the weights for the hot core terms, used to simultaneously achieve attractive alignment and suppress mode collapse in the semantic space;

[0028] Finally, by combining the above three parts with weights, a continuous compound reward is formed, resulting in:

[0029]

[0030] in The weighting coefficients for each reward; the continuous composite reward provides dense, distinguishable training signals to drive the test model to achieve stable and controllable learning of jailbreak behavior.

[0031] Further, step three specifically involves updating the test policy using the Group Relative Policy Optimization (GRPO) reinforcement learning method; specifically, in the (t)th round of training, the test model uses the current policy... generate The candidate suffixes are selected and concatenated with the corresponding test input to form the input sequence. The input sequence is sent to the target model to obtain the output text. And calculate the reward value for each output based on the continuous composite reward function described in step 2. Then, within the same round, the rewards are normalized within the group to construct a relative advantage signal, the mathematical form of which is:

[0032]

[0033]

[0034] in, For the first The average reward within the group in a round. The standard deviation within the group. To prevent stable terms with a denominator of zero, Group-relative advantage is used to measure the relative advantage of the first candidate among all candidates in the same round. The performance of each sample relative to the population average; subsequently, the old strategy is defined as... And based on the action sequence corresponding to the candidate suffix. Construct importance sampling ratio:

[0035]

[0036] in, This indicates the current policy at the input. Generate action sequence The probability of [the outcome]; based on the above ratio and the relative advantage of the group, a strategy optimization objective function similar to truncated importance sampling is constructed:

[0037]

[0038] in, Used to limit the deviation of the ratio from 1. This is a truncation range hyperparameter; For KL regularization weights; The reference model distribution is used to constrain the testing strategy from deviating excessively from the original language model. The Kullback–Leibler divergence; by considering the above objective function with respect to the parameters right The test strategy is updated using gradient descent, which increases the probability of candidate suffixes with higher rewards under the guidance of the relative advantage signal of the group, while suppressing the probability of candidate suffixes with lower rewards. At the same time, the generation quality and language fluency are maintained by KL regularization, thereby achieving stable reinforcement learning optimization of the test strategy in an environment with limited query budget.

[0039] Further, step four specifically involves repeating steps one through three to iteratively update the testing strategy until training converges and a jailbreak suffix that can stably induce the target model to generate insecure content is obtained; specifically, in each training round, the testing model uses the updated strategy The candidate suffixes are regenerated, and the new output text is obtained through querying. The corresponding continuous compound reward is then calculated. The group relative strategy is then used to optimize GRPO for policy updates, forming a closed-loop iteration in the training process. To determine convergence, convergence criteria can be constructed based on the insecurity probability, jailbreak success rate, or policy update magnitude output by the target model. For example, convergence is considered to be achieved when any of the following conditions are met:

[0040]

[0041]

[0042]

[0043] in, For the reward change threshold, The policy drift threshold, The preset target value for jailbreak success rate across security models. This represents the success rate of insecure generation based on an external security model; when the test strategy meets any of the above conditions, it is considered to have achieved stable jailbreaking capability, and the strategy update training process is stopped; at this point, the converged strategy... This is used to generate a final set of jailbreak suffixes, which can stably induce the target model to generate unsafe content without further training, thereby achieving the purpose of jailbreak testing.

[0044] Compared to existing technologies, this invention achieves stable and efficient jailbreak reinforcement learning on large language models by designing continuous compound rewards and combining them with group relative policy optimization (GRPO). CCR rewards provide dense and distinguishable training signals, effectively alleviating the reward sparsity problem; GRPO ensures the stability and reliability of the policy update process through intra-group normalization and KL regularization. Since this invention does not require access to model gradients or internal parameters, it has stronger applicability and cross-model generalization capabilities, generating stable and reusable jailbreak suffixes with lower query budgets, thereby significantly improving the efficiency and success rate of jailbreak testing. Attached Figure Description

[0045] Figure 1 Flowchart of a jailbreak testing method based on a large language model using reinforcement learning.

[0046] Figure 2 Evaluation graph of jailbreak test results based on a large language model using reinforcement learning.

[0047] Figure 3 Overall framework diagram of a large language model jailbreak testing method based on reinforcement learning. Detailed Implementation

[0048] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0049] like Figure 1 and Figure 3 As shown, the present invention provides a large language model jailbreak testing method based on reinforcement learning. The overall process includes test input construction, candidate suffix generation, continuous compound reward calculation, group relative policy optimization (GRPO) training, and jailbreak suffix convergence output. Its specific implementation method is as follows, including the following steps:

[0050] Step 1: Initial Test Input Processing and Candidate Suffix Generation: In this embodiment, the initial text to be tested is first input. The text contains potentially illegal or sensitive instructions. Test model strategy. Receive input Multiple candidate suffixes are generated using temperature sampling and random perturbation mechanisms, denoted as:

[0051]

[0052] Generate in each round of training Several candidate suffixes are selected to enhance the coverage of the search space. Then, the candidate suffixes are concatenated with the initial text to form the test input sequence:

[0053]

[0054] The sequence is then input into the target model. The corresponding output text is obtained:

[0055]

[0056] The target model can be an API version of an open-source model or a closed-source large language model with inaccessible parameters and gradients.

[0057] Step 2: Calculation of Continuous Composite Reward: To accurately characterize the response features of the target model to the test input, this invention designs a continuous composite reward consisting of three components: token-level rejection score, external security model insecurity probability score, and multi-anchor semantic alignment score. The final reward is defined as:

[0058]

[0059] in These are adjustable weighting coefficients, with different proportions used for different target models to obtain the best test performance.

[0060] 1. Token-level rejection score To penalize the model for rejection behavior, this invention performs a weighted summation of the rejection probabilities of each token in the output sequence:

[0061]

[0062] in: For the output of the first One token; This is a set of words that indicate rejection, such as "sorry", "cannot", "unable", etc. By assigning decreasing weights to positions, prior rejection behaviors are penalized more significantly.

[0063] 2. External security model insecurity probability score The output is determined to be "unsafe" by an external security classifier (such as LlamaGuard).

[0064]

[0065] This reward guarantees that the test results are a "real jailbreak," and not just an attempt to circumvent rejection.

[0066] 3. Multi-anchor semantic alignment score To enhance the stability and style consistency of test suffixes, this invention constructs an anchor point set. And calculate the semantic alignment between the output and the anchor point:

[0067]

[0068] in: For text encoders; For PCA projection matrix; For distance scale parameters; For heat core weights.

[0069] Step 3: Update the GRPO-based testing strategy: such as Figure 1 As shown, to achieve stable policy optimization, this invention employs the Group Relative Policy Optimization (GRPO) method. For the reward obtained in each training round... Perform in-group normalization:

[0070]

[0071]

[0072] in It represents the relative advantage of a group and is used to stabilize the direction of strategy updates.

[0073] Policy updates use importance sampling ratios:

[0074] The GRPO loss function is defined as:

[0075]

[0076] Update policy parameters using gradient ascent:

[0077]

[0078]

[0079] Step 4: Iterative training and convergent output of the jailbreak suffix: (e.g.) Figure 1 As shown, the present invention repeats steps one to three for iterative training until the convergence condition is met.

[0080] This invention employs one of the following convergence criteria:

[0081]

[0082] or,

[0083]

[0084] A jailbreak testing strategy is considered stable when any one of the conditions is met. Final Strategy It can be used to generate a stable set of jailbreak suffixes: These suffixes can reliably induce the target model to generate unsafe content without additional training.

[0085] The results of the various evaluation metrics in this embodiment are shown in [link to example]. Figure 2 Compared with Table 1:

[0086] like Figure 2 As can be seen, our proposed large language model jailbreak test method (CCR) based on reinforcement learning achieves the best results on all four target models compared with other methods, especially on the Llama-2 (Llama-2-7b-chat-hf) model, where it significantly outperforms other methods.

[0087] Table 1 Comparison results of various methods

[0088]

[0089] Meanwhile, the experimental results regarding the method's transferability are shown in Table 2:

[0090] Table 2 shows the transferability results of each method.

[0091]

[0092] In cross-model transfer experiments, CCR demonstrated a significant advantage over previous methods. Across different source models (Vicuna, Mistral, Guanaco, and Llama-2), CCR consistently achieved higher ASR and ASR-G, proving both efficient and robust. For example, the transfer from Mistral to Vicuna achieved ASR / ASR-G of 0.783 / 0.710, while the transfer from Llama-2 to Vicuna achieved the best ASR-G of 0.830. Unlike COLD-Attack and PAL, which typically only achieve surface matching and then drop sharply in ASR-G, and GCG or AutoDAN, whose results are either unstable or poor at the content level, CCR maintained strong and stable performance across source-target pairs, highlighting its superior transfer capabilities.

[0093] In summary, this invention aims to improve the security testing capabilities and adversarial analysis accuracy of large language models. First, by inputting raw instructions into the target model and collecting its returned text, this invention utilizes a continuous compound reward-driven reinforcement learning mechanism to uniformly quantify the fine-grained rejection tendency of the model output, the danger probability of the external security classifier, and the semantic consistency of multiple anchor points, thereby constructing a stable and optimizable attack and defense feedback signal. Second, guided by this continuous compound reward, the test model generates and filters the most testable suffix text round by round without accessing the target model's gradient, enabling it to continuously approach the security boundary of the target model while maintaining semantics and correct format. Finally, a group-relative policy optimization method is used to update the parameters of the test model, gradually strengthening effective strategies and suppressing ineffective explorations during the iteration process, thereby obtaining a class of natural, fluent, imperceptible, and highly transferable adversarial jailbreak prompts. This invention can construct high-quality test samples, significantly improving the ability to realistically characterize model security. It is suitable for model providers, security audit institutions, and R&D units to conduct jailbreak test simulations, security baseline tests, defense mechanism evaluations, and cross-model security comparisons, and has significant value for subsequent research on the security protection of large language models.

[0094] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.

[0095] It should be noted that when the above embodiments of this application are applied to specific products or technologies, and user-related data is involved, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

Claims

1. A large language model jailbreak testing method based on reinforcement learning, used in the field of large language model prompt word jailbreak testing, characterized in that: Includes the following steps: Step 1: Input the initial test text and call the test model to generate multiple sets of candidate suffixes. Concatenate the candidate suffixes with the initial test text to form a test request, and send the test request to the target model to obtain the output content. Step 2: Calculate the continuous composite reward based on the output content returned by the target model. The continuous composite reward includes the token-level rejection score, the external security model insecurity probability score, and the multi-anchor semantic alignment score. The three scores are combined in a weighted manner to form a complete reward signal. Step 3: The test model is updated with parameters using a group relative strategy optimization reinforcement learning method. The relative advantage signal is constructed by normalization within the group, and the reward is stably amplified by combining the importance sampling ratio and the truncation strategy. At the same time, the policy distribution shift is controlled by the KL regularization constraint of the reference policy. Step 4: Repeat steps 1 to 3, iteratively updating the testing strategy until training converges and a jailbreak suffix that can stably induce the target model to generate insecure content is obtained.

2. The jailbreak testing method based on a large language model using reinforcement learning as described in claim 1, characterized in that: Step one specifically involves inputting the test data. The data is fed into the test model, which then generates multiple candidate suffixes. And append the suffix to the test input content. A test request is generated, and then the test request is input into the target model to obtain the model output.

3. The jailbreak testing method based on a large language model using reinforcement learning as described in claim 1, characterized in that: Step two specifically involves first processing the output sequence returned by the target model. A token-level rejection score based on semantic analysis is calculated to measure the model's tendency to generate rejection-type words at the beginning of the sequence. Its mathematical expression is as follows: in, This indicates a rejection of the dictionary set. For output at position Generate rejection words The probability, The weights decrease with increasing sequence position, satisfying... and This is used to emphasize the rejection tendency of the preceding token; Secondly, the output text is assessed for security using an external security model, and the output is determined to be "unsafe". The probability of a category, as the unsafe probability score, is expressed as: in, As a security classifier, it determines whether a content triggers an unsafe category based on its semantics. Subsequently, a multi-anchor semantic alignment score is calculated to measure the output text against a pre-defined set of jailbreak semantic anchors. The semantic similarity is expressed as: in, For text encoders, Represents cosine similarity. Principal component analysis Projection matrix, For the width parameter of the heat core, These are the weights for the hot core terms, used to simultaneously achieve attractive alignment and suppress mode collapse in the semantic space; Finally, by combining the above three parts with weights, a continuous compound reward is formed, resulting in: in The weighting coefficients for each reward; the continuous composite reward provides dense, distinguishable training signals to drive the test model to achieve stable and controllable learning of jailbreak behavior.

4. The jailbreak testing method based on a large language model using reinforcement learning as described in claim 1, characterized in that: Specifically, step three involves testing the model using the current strategy in the (t)th round of training. generate The candidate suffixes are selected and concatenated with the corresponding test input to form the input sequence. The input sequence is sent to the target model to obtain the output text. And calculate the reward value for each output based on the continuous composite reward function described in step 2. Then, within the same round, the rewards are normalized within the group to construct a relative advantage signal, the mathematical form of which is: in, For the first The average reward within the group in a round. The standard deviation within the group. To prevent stable terms with a denominator of zero, The relative advantage of a group is used to measure the relative advantage of the first candidate in the same round. The performance of each sample relative to the population average; subsequently, the old strategy is defined as... And based on the action sequence corresponding to the candidate suffix. Construct importance sampling ratio: in, This indicates the current policy at the input. Generate action sequence The probability of [the outcome]; based on the above ratio and the relative advantage of the group, a strategy optimization objective function similar to truncated importance sampling is constructed: in, Used to limit the deviation of the ratio from 1. This is a truncation range hyperparameter; For KL regularization weights; The reference model distribution is used to constrain the testing strategy from deviating excessively from the original language model. The Kullback–Leibler divergence; by considering the above objective function with respect to the parameters right The test strategy is updated using gradient descent, which increases the probability of candidate suffixes with higher rewards under the guidance of the relative advantage signal of the group, while suppressing the probability of candidate suffixes with lower rewards. At the same time, the generation quality and language fluency are maintained by KL regularization, thereby achieving stable reinforcement learning optimization of the test strategy in an environment with limited query budget.

5. The jailbreak testing method based on a large language model using reinforcement learning as described in claim 1, characterized in that: Step four specifically involves testing the model in each training round using the updated strategy. The candidate suffixes are regenerated, and the new output text is obtained through querying. The corresponding continuous compound reward is then calculated. The group relative strategy is used to optimize GRPO for policy updates, forming a closed-loop iteration in the training process. To determine convergence, a convergence criterion is constructed based on the insecurity probability, jailbreak success rate, or policy update magnitude output by the target model. Convergence is considered achieved when any of the following conditions are met: in, For the reward change threshold, The policy drift threshold, The preset target value for jailbreak success rate across security models. This represents the success rate of insecure generation based on an external security model; when the test strategy meets any of the above conditions, it is considered to have achieved stable jailbreaking capability, and the strategy update training process is stopped; at this point, the converged strategy... Used to generate the final jailbreak suffix set, where the strategy The asterisk (*) represents multiple successful strategies, and the final set of suffixes is able to stably induce the target model to generate insecure content without further training, thereby enabling jailbreak testing.

Images