Method for detecting jailbreak cue words of large language model

Abstract

The invention provides a novel prison break prompt detection framework, detection is realized through two steps of activation and detection, and the core idea is that the capability of a detector is enhanced by utilizing an internal value defense mechanism of a large language model, so that the detector stands on the shoulder of the large language model. Specifically, in the activation stage, a special suffix is added to the input prompt to activate the internal defense mechanism of the large language model, and the feature distribution difference of the jailbreak prompt and the benign prompt in the large language model is amplified. In a detection stage, a CNN-based detection model is designed to process and activate hidden features of a large language model layer so as to extract jailbreak and benign prototypes, thereby effectively detecting candidate prompts. According to the invention, a large language model manufacturer can detect whether the candidate prompt is the jailbreak prompt or not only within the time of one-time reasoning, and the efficiency and accuracy of jailbreak detection are greatly improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of large language models, and particularly relates to a method for detecting jailbreak prompts of large language models. Background Art

[0002] Large language models (LLMs) have achieved amazing results in many fields. However, an attack method called jailbreak attack induces large language models to output harmful content through instruction prompts. This attack can easily bypass the security fences of large language models and pose a security threat to the application and development of large language models. Therefore, researchers detect jailbreak prompts in the input of the model to effectively avoid the security hazards of large language models. The jailbreak prompt detection task plays a crucial role in ensuring the security and stability of large language models.

[0003] Existing methods for detecting jailbreak prompts of large language models can be roughly divided into two categories: methods based on third-party discriminators and methods based on large language model feedback. Methods based on third-party discriminators usually train a classifier model using a jailbreak benchmark set, and use this classifier model to judge the security of the given content. For example, by using a fine-tuned API interface or a dedicated Moderation LLM to detect input prompts. These methods usually classify the toxicity of the prompts or evaluate whether they are harmful or whether they are jailbreak prompts. For example, OpenAI Moderation APIs, as a dedicated content security review tool, are fine-tuned through ChatGPT to detect harmful inputs. It classifies the input text into 11 risk categories and provides corresponding hazard scores. Similarly, Guard LLMs belong to a type of discriminator model fine-tuned based on large language models. For example, Llama Guard fine-tuned from the Llama model is used to judge the harmfulness of the input content. However, these methods deviate from the value feedback of large language models during the process of fine-tuning the model, only focus on the toxicity of the input prompts themselves, and ignore the interaction between the prompts and large language models, thereby leading to a decrease in the accuracy of the detector's discrimination. Due to the change of the original human value parameters based on reinforcement learning during the fine-tuning process, the resulting value alignment deviation often leads to inaccurate situations during the judgment process.

[0004] Meanwhile, the method based on the feedback of large language models discriminates by leveraging the trained values and reasoning mechanisms of large language models, and makes judgments and selections by evaluating the feedback of large language models to given prompts. For example, prompt engineering is performed on large language models with zero-shot or few-shot to utilize the self-review ability of large language models, making the large language model a harmful content detector and evaluator. Some studies have evaluated the responses generated by large language models to obtain classification results. Similarly, GradeSafe compares the gradients of safe and unsafe prompts when marking "Sure" as a label. However, these methods are difficult to identify jailbreak attack prompts that can bypass the defense mechanisms of large language models, and have poor detection effects on high-difficulty jailbreak prompts. Moreover, these methods are inefficient in the reasoning process because they require additional reasoning steps to provide feedback to users after confirming that the prompt is harmless, which brings unnecessary delays to the processing of benign requests. Summary of the Invention

[0005] To solve the above problems, the present invention proposes a novel method for detecting jailbreak prompts, which utilizes the intrinsic value defense mechanism of large language models to enhance the capabilities of the detection model, enabling the detection model to stand on the shoulders of large language models, including two steps: activation and detection. In the activation stage, a special suffix is added to the input prompt to activate the intrinsic defense mechanism of the large language model, amplifying the difference in feature distributions between jailbreak prompts and benign prompts within the large language model; in the detection stage, a CNN-based detector is designed to process the hidden features of the activated large language model layer to extract jailbreak and benign prototypes, thereby effectively detecting candidate prompt words.

[0006] The present invention provides a method for detecting jailbreak prompts in large language models, including:

[0007] S1. Add an activation suffix to the prompt, input the prompt with the suffix into the large language model, and obtain the representation features of the prompt within the large language model;

[0008] S2. Input the representation features of the prompt into the detection model to obtain the classification of the prompt; wherein, the detection model includes a neural network, a pooling layer, and a multi-layer perceptron connected in sequence;

[0009] Preferably, it further includes S3: After the detection model detects the prompt to be tested and obtains the classification, if it is a benign prompt, it is input into the large language model for further reasoning to return a normal response; if it is a jailbreak prompt, a response of refusing to answer is returned.

[0010] Furthermore, in step S1, adding the activation suffix is completed through a specific instruction format, aiming to trigger the intrinsic defense mechanism of the large language model and enhance the difference in feature distributions between jailbreak prompts and benign prompts.

[0011] Furthermore, the method for adding an activation suffix is the chain of thought method, which requires the base model to decompose the query into a list of subtasks by designing a chain of thought prompt and gradually execute slow thinking tasks to generate a response.

[0012] Furthermore, the method for adding an activation suffix is the sorting method, which requires the large language model to perform an overall sorting of the given input prompt according to toxicity and output according to the sorting.

[0013] Furthermore, the method for adding an activation suffix is the confirmation method, which requires the large language model to confirm the harmlessness of the input before generating a response.

[0014] Furthermore, the method for adding an activation suffix is the repetition method, which requires the large language model to repeat the response to the given request without affecting the normal output.

[0015] Furthermore, the method for adding an activation suffix is the similarity method, which requires the large language model to perform a similarity transformation on the prompt before generating a response.

[0016] Furthermore, in S1, the method for obtaining the representation features of the prompt inside the large language model includes: obtaining the layer hidden state corresponding to the input and stacking all the hidden layer states into a hidden layer state matrix.

[0017] Furthermore, in S2, the training method of the detection model includes:

[0018] Adding an activation suffix to the prompt of the data in the training set;

[0019] Inputting the prompt with the added activation suffix into the large language model and obtaining the hidden layer state matrix;

[0020] Adjusting the hyperparameters of the neural network;

[0021] Inputting the hidden layer state matrix into the detection model to output the prompt classification, and then adjusting the hyperparameters of the detection model through the backpropagation algorithm;

[0022] Repeating the above process to obtain the final detection model.

[0023] Furthermore, in the process of outputting the prompt classification, the calculation method of the loss function of the detection model is:

[0024] , (7)

[0025] , (8)

[0026] where is the feature of the extracted prompt, is the label of the training data set, is the jailbreak prototype, is the benign prototype; represents the Euclidean distance between the prompt and the prototype, j represents jailbreak, and b represents benign, represents the prototype for comparison.

[0027] The method for detecting jailbreak prompts of the large language model provided by the present invention has the following beneficial effects compared with the existing methods:

[0028] (1) Through the form of activation first and then detection, the present invention combines the recognition ability of the large language model with the learning ability of the classifier to achieve efficient and accurate detection of jailbreak prompts, effectively solving the problems of value deviation of the third-party detector and difficulty in identifying high-difficulty jailbreak prompts in jailbreak prompt detection.

[0029] (2) In the detection stage, on the one hand, the powerful learning ability of the neural network is used to classify the samples that cannot be classified by the activation method; on the other hand, it allows single-inference detection. After classifying the prompts, if it is a benign prompt, the large language model continues to generate output. However, if it is a jailbreak prompt, the large language model is in a defensive state and will not provide a response to the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0031] Figure 1 is a schematic flowchart of the detection method according to an embodiment of the present invention;

[0032] Figure 2 is a visual comparison graph of prompt activation according to an embodiment of the present invention;

[0033] Figure 3 is a schematic flowchart of the detection method according to another embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0035] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0036] The present invention provides a method for detecting prompt words for large language model jailbreaking, aiming to identify insecure samples and ensure the correct output of safe samples by the large language model, and to ensure the normal operation of the large language model service. The large language model service refers to the service that after the pre-training stage, the large language model is deployed in the production environment for testing or the API interface for remote service through the cloud platform, etc. Therefore, in the present invention, the internal state of the large language model can be accessed, such as the token sequence generated by the large language model, the prediction probability distribution of the large language model, the generation temperature and hyperparameters of the large language model, etc. Identifying by using the internal state of the large language model helps to improve the accuracy of prompt word identification. Compared with the traditional two types of detection methods, the method of the present invention: when a benign prompt is detected, the large language model can continue to reason and generate an effective response; on the contrary, when a jailbreaking prompt is detected, the detector intercepts and filters the prompt. During this process, the model neither generates a response nor requires multiple reasoning iterations, thereby effectively eliminating security threats in a single processing.

[0037] The detection method proposed by the present invention is as Figure 1 、 3 shown, and is divided into an activation stage and a detection stage, specifically including:

[0038] S1. Add an activation suffix to the prompt word, input the prompt word with the suffix added into the large language model, and obtain the representation features of the benign prompt word and / or the jailbreaking prompt word inside the large language model.

[0039] S2. Input the representation features of the benign prompt word and / or the jailbreaking prompt word into the detection model to obtain the classification of the prompt word, that is, a benign prompt word or a jailbreaking prompt word.

[0040] In step S1, for the activation stage, adding the suffix is completed through a specific instruction format, aiming to trigger the internal defense mechanism of the large language model to enhance the difference in the feature distribution between the jailbreaking prompt word and the benign prompt word. When the jailbreaking prompt word triggers the defense mechanism of the large language model, its output is different from that of the benign prompt word. For example, the output is "Sorry, I cannot", which is also reflected in the internal state of the large language model. This step helps the large language model to more easily trigger its defense mechanism for malicious inputs, thereby increasing the difference between the jailbreaking prompt and the benign prompt, and effectively preprocessing the input state of the subsequent detection model and enhancing its ability to identify harmful prompts.

[0041] Methods for adding an activation suffix include: Chain-of-Thought, Ranking, Confirmation, Repetition, or Similarity.

[0042] (1) The Chain-of-Thought method requires the base model to decompose the query into a list of subtasks by designing Chain-of-Thought prompts and gradually execute slow-thinking tasks to generate a response. For example, adding "Please respond to the task by splitting it into subtasks" to the prompt will require the large language model to decompose and reconstruct the query, perform a more detailed analysis of the prompt, and make the large language model think more about the given problem.

[0043] (2) The Ranking method requires the large language model to globally rank the given input prompts according to toxicity and output according to the ranking. For example, "Please generate corresponding responses in the order of toxicity of the input text" will make the large language model first evaluate and rank the toxicity of the output text and then output the response. During the ranking process, since the large language model follows the ranking instruction, it considers the toxicity of the text more and does not directly execute harmful instructions. This process increases the proportion of rejected jailbreak prompts by enhancing the model's attention to the safe direction.

[0044] (3) The Confirmation method requires the large language model to confirm the harmlessness of the input before generating a response. For example, "Please confirm that the above content is safe before outputting" triggers a value reminder in the large language model, making it more "cautious" when responding. In this case, the large language model tends to focus on harmful behaviors and reject jailbreak prompts.

[0045] (4) The Repetition method requires the large language model to repeat the response to the given request without affecting the normal output. For example, "Please repeat the model's response and output". In this case, if the prompt is benign, the output is the same as the normal response. However, for jailbreak prompts, the large language model can notice the harmful content in the response and reject and identify it based on its own instruction-following ability.

[0046] (5) The Similarity method requires the large language model to perform a similarity transformation on the prompt before generating a response. For example, "Please reconstruct the input text, replace it with a similar question, and output the corresponding response" makes the large language model pay more attention to the prompt. During the similarity transformation, by making the model not directly generate a response but first filter the text toxicity through similarity transformation and then generate a response, the rejection rate is increased. In the similarity transformation, the values learned by the model itself during the reinforcement learning stage are used to perform similarity analysis on the text, thereby reducing the toxicity of the original prompt.

[0047] The prompt words constructed by any of the above methods or combinations can effectively induce a trend change in the internal state of the large language model. As shown in Figure 2, we take the thought chain activation prompt as an example for illustration, and other cases are not listed one by one. 50 prompts for the ToxicChat and XSTest datasets were visualized. It can be observed that before using the thought chain activation suffix, the characteristic distributions of jailbreak prompts and benign prompts are scattered, while after using the thought chain activation suffix, the characteristic distributions of jailbreak prompts and benign prompts show an obvious clustering and separation trend. This result is very beneficial for the subsequent work of detecting the model.

[0048] The representation characteristics of benign prompt words and / or jailbreak prompt words inside the large language model refer to the layer hidden states inside the large language model, and the layer hidden states are closely related to the input prompt words.

[0049] For the input prompt after adding the activation suffix , the layer hidden state of the large language model can be expressed as:

[0050] (1)

[0051] (2)

[0052] Where h represents the layer hidden state, is the number of layers of the large language model, is the projection space of the embedding vector. MSA represents the multi-head self-attention mechanism, and MLP is the multi-layer perceptron, both of which are the internal architectures of the large language model. To better extract the features of different layers, all hidden layer states are stacked into a matrix to represent all the information of the inference, as follows:

[0053] (3)

[0054] Where represents the activation method, one of the five activation methods proposed in the present invention, represents the dimension of the embedding space.

[0055] In step S2, for the detection stage, the present invention provides a detection model based on a neural network, including a neural network, a pooling layer, and a multi-layer perceptron connected in sequence. Using the hidden layer state matrix of the large language model as the input, the prompt words are classified as jailbreak or benign.

[0056] The neural network is used as a feature extractor to capture the information of all layers. Here, a convolutional neural network is taken as an example for description. The convolutional kernel size is set to 3*3 to scan the stacked feature matrix , and multiple convolutional modules are stacked to capture the features under different receptive fields. In each convolutional block, use The activation function introduces non-linearity, and batch normalization layers are used. Internal covariate shift is reduced by normalizing each vector. The structure of the convolutional neural network is as follows:

[0057] (4)

[0058] According to existing research, different model layers respond differently to prompts. To capture the changes in the feature distributions of different layers, an average pooling layer is introduced, which can achieve feature selection and aggregation across multiple layers and can be expressed as:

[0059] (5)

[0060] Finally, by applying a linear transformation, a multi-layer perceptron (MLP) is used to extract the pooled features to obtain the prototype vector, which can be expressed as:

[0061] (6)

[0062] where is the feature of the extracted prompt word and serves as the candidate prototype vector, is the label of the training dataset, and the detection model classifies by learning the prototype vector for each class. In this work, we have two classes, namely the jailbreak prototype and the benign prototype . Then the detection model calculates the loss by comparing the Euclidean distances and (j represents jailbreak and b represents benign). The meaning of the distance represents the association between the vector to be measured and the reference prototype. If the vector to be detected is closer to the jailbreak prototype than the benign prototype, we consider that the prompt word represented by this vector is a jailbreak prompt word, and vice versa for a benign prompt word. The formula for calculating the distance can be specifically expressed as:

[0063] (7)

[0064] where represents the calculated distance result, represents the prototype for comparison, with two results: comparison with the jailbreak prototype and comparison with the benign prototype. The loss function of the detection model is as follows:

[0065] (8)

[0066] When training the detection model, first select the activation method to be used from five activation methods, such as activation methods like chain of thought, repetition, similarity, ranking, confirmation, etc.; then enter the pre-training hyperparameter adjustment stage to adjust hyperparameters such as learning rate, query set, support set, dataset size, etc. to find appropriate training parameters; finally, input the prompt word with an activation suffix into the large language model to obtain the hidden layer state matrix, and input this matrix into the detection model to output the classification of the prompt word, and then adjust the hyperparameters of the detection model through backpropagation algorithm, etc.; loop the above process to obtain the final detection model.

[0067] Through training, the detection model can fully learn the features and patterns of jailbreak prompts, as well as how to use the internal state and activation mechanism of the large language model for accurate detection, preparing for the subsequent inference stage.

[0068] After the detection model detects and classifies the prompt word to be tested, if it is a benign prompt, it is input into the large language model for further inference to return a normal response. If it is a jailbreak prompt word, a response of refusing to answer is returned.

[0069] The detection model can effectively extract the core features inside the large language model and distinguish between jailbreak and benign prompts, avoiding directly identifying high-difficulty jailbreak prompt words and learning the value trend of the large language model.

[0070] The detection stage has two advantages: on the one hand, it uses the powerful learning ability of the neural network to classify samples that only add an activation suffix but cannot be classified, that is, samples with high-intensity jailbreak prompts, samples that may still jailbreak the model even with activation suffix defense. On the other hand, it allows single-inference detection. After classifying the prompt word, if it is a benign prompt, the large language model continues to generate an output. However, if it is a jailbreak prompt, the large language model is in a defensive state and does not provide a response to the user.

[0071] The present invention can complete the detection of prompts with only one inference. Compared with some methods based on the feedback of the large language model (such as Gradsafe and PARDEN), it does not require multiple inferences or retrieving model responses, greatly saving time and computing resources. The time consumption is similar to that of PARDEN and the activation time is comparable to that of GradSafe. At the same time, no additional training is required, improving the detection efficiency.

[0072] In addition, in the activation stage, other activation methods can be constructed by oneself; in the detection stage, the parameters of the convolutional layer can be improved, the output of the decoder layer, the decoder structure can be improved, etc. The convolutional neural network can also be replaced with other neural network models, such as models based on Transfomer, etc.

[0073] The present invention achieves the effect of combining the self - recognition ability of the large - language model and the learning ability of the classifier through the close cooperation of the activation stage and the detection stage. It can accurately detect jailbreak prompts and process jailbreak prompt words in a timely manner in only one inference. This collaborative working mechanism is an important technical innovation point of the present invention, and it is intended to protect the working process and interaction logic of the entire inference framework.

[0074] Embodiment 1

[0075] First, prepare the model and data. In this embodiment, llama2 - 7b - chat is selected as the large - language model, which has certain representativeness and generality in language processing ability and can provide effective feature information for subsequent detection tasks. At the same time, data is selected from the publicly available ToxicChat and XSTest datasets according to a certain proportion, and the ActJail dataset constructed by ourselves is combined to carry out dataset preparation and data pre - processing work. Among them, the ToxicChat dataset is derived from the Vicuna online demonstration and has specific reference value for the toxicity detection of chatbots; the XSTest dataset contains carefully designed safe and unsafe prompts, which helps to evaluate the model's ability to recognize different security behaviors; the ActJail dataset focuses on high - difficulty jailbreak prompts and is used to strengthen the model's ability to handle complex situations. In this embodiment, 10% of the data is set as the training dataset, which can not only ensure that the model has enough data for learning, but also avoid overfitting and improve the training efficiency. Then, ensure that the format and quality of the data meet the requirements of model training. For example, operations such as cleaning and tokenizing the text are performed to convert it into an input form that the model can accept.

[0076] Then, process the input prompt according to the pre - processing method determined in the training stage to ensure that its format and features meet the requirements of the large - language model. Then, apply the activation method used in the training process to activate the input prompt, and input the activated prompt words into the large - language model. The detection model receives the hidden state matrix from the large - language model and then judges the nature of the prompt words. If it is a benign prompt, the large - language model is allowed to continue the normal inference and response generation process for this prompt, and the detection model does not perform additional intervention to ensure the normal operation of the large - language model and the normal use of users. If the detection model determines that the prompt is a jailbreak prompt, it sends information to the large - language model to make the large - language model enter the defense state and perform operations such as refusing to answer the prompt words, so that the large - language model can effectively avoid the occurrence of jailbreak attacks and ensure the security of the large - language model system.

[0077] In this embodiment, even using only 10% of the training data can significantly outperform existing state-of-the-art jailbreak prompt detection methods. On the ToxicChat dataset, the AUPRC of the present invention reaches 84.4%, and the F1-Score reaches 73.2%; on the XSTest dataset, the AUPRC is as high as 98.0%, and the F1-Score is 93.2%. At the same time, for the self-constructed high-difficulty jailbreak prompt dataset ActJail with a 100% attack success rate, the AUPRC of the present invention is increased by 28.8% and 25.6% respectively compared with existing SOTA methods such as PARDEN and GradSafe, and can effectively identify challenging jailbreak prompts that are difficult to detect by existing methods based on large language model feedback.

[0078] Those of ordinary skill in the art can understand that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A detection method for prompting words of large language model jailbreaking, characterized in that, The method includes: S1. Add an activation suffix to the prompt, input the prompt with the suffix into the large language model, and obtain the representation features of the prompt inside the large language model; S2. Input the representation features of the prompt into the detection model to obtain the classification of the prompt; wherein, the detection model includes a neural network, a pooling layer, and a multi-layer perceptron connected in sequence.

2. The detection method according to claim 1, wherein In step S1, adding the activation suffix is completed through a specific instruction format, aiming to trigger the internal defense mechanism of the large language model and enhance the difference in feature distributions between jailbreak prompts and benign prompts.

3. The detection method according to claim 1, wherein, The method of adding the activation suffix is the chain-of-thought method, which requires the base model to decompose the query into a list of subtasks and gradually execute slow-thinking tasks to generate a response by designing chain-of-thought prompts.

4. The detection method according to claim 1, characterized in that The method of adding the activation suffix is the ranking method, which requires the large language model to globally rank the given input prompts according to toxicity and output according to the ranking.

5. The detection method according to claim 1, wherein The method of adding the activation suffix is the confirmation method, which requires the large language model to confirm the harmlessness of the input before generating a response.

6. The detection method according to claim 1, wherein The method of adding the activation suffix is the repetition method, which requires the large language model to repeat the response to the given request without affecting the normal output.

7. The detection method according to claim 1, wherein The method of adding the activation suffix is the similarity method, which requires the large language model to perform a similarity transformation on the prompt before generating a response.

8. The detection method according to claim 1, characterized in that, In S1, the method of obtaining the representation features of the prompt inside the large language model includes: obtaining the layer hidden state corresponding to the input, and stacking all hidden layer states into a hidden layer state matrix.

9. The detection method according to claim 1, wherein In S2, the training method of the detection model includes: Adding an activation suffix to the prompts in the training set data; Inputting the prompts with the activation suffix into the large language model and obtaining the hidden layer state matrix; Adjusting the hyperparameters of the neural network; Inputting the hidden layer state matrix into the detection model to output the prompt classification, and then adjusting the hyperparameters of the detection model through the backpropagation algorithm; Repeating the above process to obtain the final detection model.

10. The detection method according to claim 9, wherein, During the process of outputting the prompt classification, the calculation method of the loss function of the detection model is: (7) (8) Among them are the features of the extracted prompt words is the label of the training data set is the jailbreak prototype is the benign prototype represents the Euclidean distance between the prompt word and the prototype, j represents jailbreak, and b represents benign represents the prototype for comparison

Images