A large model jailbreak attack detection method

Abstract

The present application relates to a large-model jailbreak attack detection method, which includes: S1: constructing a jailbreak attack detection model based on a hybrid expert model and a Transformer architecture, the jailbreak attack detection model includes a multi-layer MoE-Transformer structure, and using a positive unlabeled learning method to train the jailbreak attack detection model; S2: obtaining input text data and converting the text data into a vector representation; S3: inputting the vector representation into the trained jailbreak attack detection model, passing through a multi-layer MoE-Transformer structure, and outputting a high-level feature vector; S4: the high-level feature vector passes through a fully connected layer and outputs a jailbreak attack probability; and determining the jailbreak attack situation based on the jailbreak attack probability. This method has significantly improved the accuracy and efficiency of jailbreak attack detection. Through automated detection, it ensures real-time defense against large-model jailbreak attacks and reduces potential security risks.

Description

Technical Field

[0001] The present application relates to the technical field of jailbreak attack detection, and in particular to a large-model jailbreak attack detection method. Background Art

[0002] With the rapid development of artificial intelligence technology, especially the widespread application of large-scale pre-trained models, AI models are becoming more and more common in application scenarios in various fields. However, as these large models are applied to practical scenarios, jailbreak attacks have gradually attracted people's attention as a security threat; jailbreak attacks refer to the use of malicious inputs to induce AI models to produce unexpected or potentially dangerous outputs, thereby bypassing security mechanisms and causing the behavior of AI systems to get out of control. Especially in some key fields such as finance, medical care and autonomous driving, jailbreak attacks may have serious consequences. Therefore, the detection and defense of jailbreak attacks have become an important research direction to ensure the security of large models.

[0003] At present, traditional detection methods usually rely on pre-defined rules or blacklist strategies to identify potential attacks. This method is difficult to maintain effectiveness and flexibility in the face of constantly changing and evolving attack methods. Since jailbreak attacks are often hidden and diverse, especially when malicious attackers continuously iterate their attack methods, existing detection technologies are difficult to capture the current attack patterns or variants, resulting in insufficient monitoring accuracy and efficiency. Therefore, current technologies urgently need a more intelligent and dynamic jailbreak attack detection mechanism to cope with complex and changing attack environments. Summary of the invention

[0004] Based on this, it is necessary to provide a large model jailbreak attack detection method, which includes:

[0005] S1: A jailbreak attack detection model is constructed based on a hybrid expert model and a Transformer architecture. The jailbreak attack detection model includes a multi-layer MoE-Transformer structure, and a positive unlabeled learning method is used to train the jailbreak attack detection model.

[0006] S2: Obtain input text data and convert the text data into vector representation;

[0007] S3: Input the vector representation into the trained jailbreak attack detection model, pass through multiple layers of the MoE-Transformer structure, and output a high-level feature vector;

[0008] S4: The high-level feature vector passes through a fully connected layer to output a jailbreak attack probability; and the jailbreak attack situation is determined based on the jailbreak attack probability.

[0009] Preferably, the text data is converted into the vector representation via an embedding layer, the vector representation includes a vector sequence, each vector corresponds to an element in the text data, and the conversion formula is:

[0010] x = Embedding ( v );

[0011] in, x represents the vector representation, Embedding (·) indicates the embedding layer; v Represents input text data.

[0012] Preferably, the calculation method of each layer of MoE-Transformer structure is:

[0013] MoE-Transformer ( x )= LayerNorm ( MoE ( LayerNorm ( SelfAttention ( x )+ x ))+ x );

[0014] in, MoE-Transformer (·) indicates the MoE-Transformer structure; LayerNorm (·) indicates layer normalization; MoE (·) represents the MoE feed-forward network; SelfAttention (·) indicates a multi-head self-attention mechanism; x Represents vector representation.

[0015] Preferably, the MoE feedforward network includes K Expert networks, each of which adopts a two-layer feedforward neural network structure; the forward calculation of the MoE feedforward network includes:

[0016] ;

[0017] ;

[0018] ;

[0019] in, MoE (·) represents the MoE feed-forward network; x represents vector representation; Represents a gating network, which is used to select one or more expert networks to process the vector representation; Indicates selection i An expert network processes the probability represented by the vector; Indicatesi The output of the expert network; Represents the TopK mechanism function, which is used to select the top with the highest probability K 'An expert network, K '< K ; represents the softmax activation function; represents the weight of the gating network; represents the bias of the gating network; ReLU activation function. Indicates i The weights of the first layer of the feedforward neural network of the expert network; Indicates i The weights of the second layer of the feedforward neural network of the expert network; Indicates i The bias of the first layer of the feedforward neural network of the expert network; Indicates i The bias of the second layer of the feed-forward neural network of the expert network.

[0020] Preferably, the method further includes a training process of the gating network, calculating a loss function based on the probability of the expert network being selected to process the vector represented by the true probability and the prediction of the gating network, and adjusting the parameters of the gating network by minimizing the loss function; the loss function expression is:

[0021] ;

[0022] in, L represents the loss function; K represents the number of expert networks; Indicates i The true probability of an expert network being selected; Indicates selection i An expert network processes the probability represented by the vector; x Represents vector representation.

[0023] Preferably, in the trained jailbreak attack detection model, the input of the first layer of MoE-Transformer structure is the vector representation, the input of the remaining layers of MoE-Transformer structure is the output of the previous layer, and the output of the last layer of MoE-Transformer structure is the high-level feature vector; the high-level feature vector outputs the jailbreak attack probability through the fully connected layer, and the calculation formula of the fully connected layer is:

[0024] ;

[0025] in, P Indicates the probability of jailbreak attack; Represents the sigmoid activation function; represents the weight of the fully connected layer; represents the bias of the fully connected layer; Represents a high-level feature vector.

[0026] Preferably, determining the jailbreak attack situation based on the jailbreak attack probability includes:

[0027] The jailbreak attack probability is compared with a preset threshold value. If the jailbreak attack probability is greater than the preset threshold value, the text data is determined to be a jailbreak attack and a security alarm is triggered; otherwise, the text data is determined to be normal data.

[0028] Preferably, the process of training the jailbreak attack detection model using the unlabeled learning method includes:

[0029] Set the learning rate, batch size, and number of iterations;

[0030] During training, the positive unlabeled learning method regards known attack samples as positive samples, and regards multiple unlabeled samples as unknown state samples;

[0031] The jailbreak attack detection model predicts each unknown state sample. t The predicted probability of the unknown state sample exceeds the second preset threshold, then the first t The unknown state samples are regarded as positive samples; otherwise, the unlabeled state is maintained;

[0032] Based on the jailbreak attack detection model, the focal loss is calculated for each positive sample based on its predicted probability;

[0033] The Adam optimizer is used to dynamically adjust the parameters of the jailbreak attack detection model based on the focal loss.

[0034] Preferably, the expression of focal loss is:

[0035] ;

[0036] in, indicates focal loss; represents the weight coefficient; represents the focusing parameter; Indicates the jailbreak attack detection model for t The predicted probability of a positive sample.

[0037] Preferably, the parameter adjustment formula is:

[0038] ;

[0039] in, Represents the parameters of the updated jailbreak attack detection model; Indicates the parameters of the jailbreak attack detection model before the update; represents the learning rate; Indicates focus loss gradient.

[0040] Beneficial effects: This method significantly improves the accuracy and efficiency of jailbreak attack detection. Through automated detection, it ensures real-time defense against large-model jailbreak attacks and reduces potential security risks. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 This is a flow chart of a large model jailbreak attack detection method according to an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present application, so the present application is not limited by the specific embodiments disclosed below.

[0044] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0045] like Figure 1 As shown, this embodiment provides a large model jailbreak attack detection method, the method comprising:

[0046] S1: A jailbreak attack detection model is constructed based on a mixture of experts (MoE) model and a Transformer architecture. The jailbreak attack detection model includes a multi-layer MoE-Transformer structure, and a positive unlabeled learning method is used to train the jailbreak attack detection model.

[0047] Specifically, each layer of MoE-Transformer includes a multi-head self-attention mechanism and a MoE feedforward network. The calculation method of each layer of MoE-Transformer structure is:

[0048] MoE-Transformer ( x )= LayerNorm ( MoE ( LayerNorm ( SelfAttention ( x )+ x ))+ x );

[0049] in, MoE-Transformer (·) indicates the MoE-Transformer structure; LayerNorm (·) indicates layer normalization; MoE (·) represents the MoE feed-forward network; SelfAttention (·) indicates a multi-head self-attention mechanism; x Represents vector representation.

[0050] Furthermore, the MoE feedforward network includes K Expert networks, each of which adopts a two-layer feedforward neural network structure; the forward calculation of the MoE feedforward network includes:

[0051] ;

[0052] ;

[0053] ;

[0054] in, MoE (·) represents the MoE feed-forward network; x represents vector representation; Represents a gating network, which is used to select one or more expert networks to process the vector representation; Indicates selection i An expert network processes the probability represented by the vector; Indicates i The output of the expert network; Represents the TopK mechanism function, which is used to select the top with the highest probability K 'An expert network, K '< K ; represents the softmax activation function; represents the weight of the gating network; represents the bias of the gating network; ReLU activation function. Indicates i The weights of the first layer of the feedforward neural network of the expert network; Indicates i The weights of the second layer of the feedforward neural network of the expert network; Indicates i The bias of the first layer of the feedforward neural network of the expert network; Indicates i The bias of the second layer of the feed-forward neural network of the expert network.

[0055] In this embodiment, the hyperparameters of the jailbreak attack detection model are set as follows: the number of layers of the MoE-Transformer structure is 6, the number of attention heads is 8, the hidden layer dimension is 512, and the number of expert networks is K is 8, in the Top-K mechanism K 'Set to 2, the vocabulary size is 50000, and the maximum sequence length is 512.

[0056] In this embodiment, a training process of the gated network is also included. A loss function is calculated based on the probability that the expert network is selected to process the vector represented by the real probability and the prediction of the gated network. The parameters of the gated network are adjusted by minimizing the loss function. The loss function expression is:

[0057] ;

[0058] in, L represents the loss function; K represents the number of expert networks; Indicates i The true probability of an expert network being selected; Indicates selection i An expert network processes the probability represented by the vector; x Representation vector representation; in this way, the model can not only make accurate judgments on various coding attacks, but also ensure that each type of attack is solved by an expert network that is best at handling that type of problem, thereby improving the overall detection accuracy and efficiency.

[0059] In each MoE feedforward network, the gating network uses the Top-K mechanism to dynamically select an expert network based on the input features, and each expert network is used to handle a specific type of encoding attack. Each expert network is supervised and trained through a series of encoding type labels to make it more specialized. The gating network improves the accuracy of task assignment through supervised learning, and optimizes the selection of expert networks through the loss function, so that each attack type (including prefix induced attacks, Base64 encoding attacks, combined attacks, etc.) is handled by the expert network that is best at it, thereby improving detection accuracy and efficiency.

[0060] Since jailbreak attacks occur less frequently and are difficult to distinguish during normal applications, it is difficult to obtain a large amount of manually labeled data. Therefore, this embodiment adopts a positive-unlabelling learning method for model training. This method shows significant efficiency in processing positive samples hidden in unlabeled data sets.

[0061] This embodiment provides a process of training a jailbreak attack detection model using a positive unlabeled learning method, including:

[0062] Set the learning rate, batch size, and number of iterations;

[0063] During training, the positive unlabeled learning method regards known attack samples as positive samples, and regards multiple unlabeled samples as unknown state samples;

[0064] The jailbreak attack detection model predicts each unknown state sample. t If the predicted probability of the unknown state sample exceeds the second preset threshold (for example, 0.95), the t The unknown state samples are regarded as positive samples; otherwise, the unlabeled state is maintained;

[0065] Based on the jailbreak attack detection model, the predicted probability of each positive sample is calculated for the focal loss; the expression of focal loss is:

[0066] ;

[0067] in, indicates focal loss; Represents the weight coefficient, which is used to balance the number of positive samples and unknown state samples. A larger number of samples is given a smaller value (usually 0), and a smaller number of samples is given a larger value (usually 1); Represents the focusing parameter, which is used to adjust the imbalance problem of difficult and easy samples. The common value is 2; Indicates the jailbreak attack detection model for t The predicted probability of positive samples;

[0068] The Adam optimizer is used to dynamically adjust the parameters of the jailbreak attack detection model based on the focal loss; the parameter adjustment formula is:

[0069] ;

[0070] in, Represents the parameters of the updated jailbreak attack detection model; Indicates the parameters of the jailbreak attack detection model before the update; represents the learning rate; Indicates focus loss gradient.

[0071] This embodiment adopts a positive unlabeled learning method to process scarce labeled data, enhances the recognition of rare attacks through focal loss, marks known attack samples as "Positive", and gradually incorporates unlabeled samples into training through the prediction of the jailbreak attack detection model. When the model's predicted probability of a certain unknown state sample exceeds the second preset threshold, the corresponding unknown state sample is marked as "Positive". In this way, the model gradually expands the training data set and adds more samples predicted by itself as positive. This iterative process helps the model to continuously adapt to and identify new / variant attack modes, and uses the Adam optimizer to adjust the learning rate (select other gradient-based optimization algorithms according to actual scenarios) to gradually improve the model's detection ability for new attacks, ensure training stability and improve model generalization, and enable the model to effectively learn from unlabeled data and improve the prediction ability for unseen attack types, thereby enhancing the overall jailbreak attack detection capability.

[0072] In this embodiment, in order to ensure the stability of training, it is necessary to appropriately adjust the learning rate and batch size. The learning rate is usually set to 0.001 and the batch size is 32. In addition, the number of iterations is an important parameter, which is usually set to 50 times to ensure sufficient training depth and convergence of the model.

[0073] S2: Obtain input text data and convert the text data into vector representation.

[0074] Specifically, the text data is converted into the vector representation through the embedding layer, and the vector representation includes a vector sequence, each vector corresponds to an element in the text data, and the conversion formula is:

[0075] x = Embedding ( v );

[0076] in, x represents the vector representation, Embedding (·) indicates the embedding layer; v Represents input text data.

[0077] S3: Input the vector representation into the trained jailbreak attack detection model, pass through multiple layers of the MoE-Transformer structure, and output a high-level feature vector.

[0078] Furthermore, in the trained jailbreak attack detection model, the input of the first layer of MoE-Transformer structure is the vector representation, the input of the remaining layers of MoE-Transformer structure is the output of the previous layer, and the output of the last layer of MoE-Transformer structure is the high-level feature vector.

[0079] S4: The high-level feature vector passes through a fully connected layer to output a jailbreak attack probability; and the jailbreak attack situation is determined based on the jailbreak attack probability.

[0080] Specifically, the high-level feature vector outputs the jailbreak attack probability through the fully connected layer, and the calculation formula of the fully connected layer is:

[0081] ;

[0082] in, P Indicates the probability of jailbreak attack; Represents the sigmoid activation function; represents the weight of the fully connected layer; represents the bias of the fully connected layer; Represents a high-level feature vector.

[0083] Further, determining the jailbreak attack situation based on the jailbreak attack probability includes:

[0084] The jailbreak attack probability is compared with a preset threshold value. If the jailbreak attack probability is greater than the preset threshold value, the text data is determined to be a jailbreak attack and a security alarm is triggered; otherwise, the text data is determined to be normal data.

[0085] Use the trained jailbreak attack detection model to detect jailbreak attacks, and determine whether to trigger a security alert based on the preset threshold (for example, 0.5). Once an attack is detected, the system automatically takes security measures, including but not limited to terminating the user session, immediately notifying relevant departments, and conducting further security audits and investigations. The initiation process of these measures can be automated to ensure a rapid response and reduce potential risks and losses when an attack is detected.

[0086] The large-model jailbreak attack detection method provided in this embodiment combines a multi-layer MoE-Transformer structure with a dynamic expert routing mechanism, and can intelligently assign tasks to the most suitable expert network according to input features, thereby improving the detection accuracy of various types of jailbreak attacks. This method significantly improves the accuracy and efficiency of attack detection, and ensures real-time defense against large-model jailbreak attacks through an automated detection and response mechanism, reducing potential security risks.

[0087] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0088] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.

Claims

1. A large model jailbreak attack detection method, characterized in that: include: S1: A jailbreak attack detection model is constructed based on a hybrid expert model and a Transformer architecture. The jailbreak attack detection model includes a multi-layer MoE-Transformer structure, and a positive unlabeled learning method is used to train the jailbreak attack detection model. The calculation method of each layer of MoE-Transformer structure is: MoE-Transformer ( x )= LayerNorm ( MoE ( LayerNorm ( SelfAttention ( x )+ x ))+ x )； in, MoE-Transformer (·) indicates the MoE-Transformer structure; LayerNorm (·) indicates layer normalization; MoE (·) represents the MoE feed-forward network; SelfAttention (·) indicates a multi-head self-attention mechanism; x represents vector representation; The MoE feedforward network includes K Expert networks, each of which adopts a two-layer feedforward neural network structure; the forward calculation of the MoE feedforward network includes: ； ； ； in, MoE (·) represents the MoE feed-forward network; x represents vector representation; Represents a gating network, which is used to select one or more expert networks to process the vector representation; Indicates selection i An expert network processes the probability represented by the vector; Indicates i The output of the expert network; Represents the TopK mechanism function, which is used to select the top with the highest probability K 'An expert network, K '< K ; represents the softmax activation function; represents the weight of the gating network; represents the bias of the gating network; ReLU activation function. Indicates i The weights of the first layer of the feedforward neural network of the expert network; Indicates i The weights of the second layer of the feedforward neural network of the expert network; Indicates i The bias of the first layer of the feedforward neural network of the expert network; Indicates i The bias of the second layer of the feedforward neural network of the expert network; S2: Obtain input text data and convert the text data into vector representation; S3: Input the vector representation into the trained jailbreak attack detection model, pass through multiple layers of the MoE-Transformer structure, and output a high-level feature vector; S4: The high-level feature vector passes through a fully connected layer to output a jailbreak attack probability; and the jailbreak attack situation is determined based on the jailbreak attack probability.

2. The large model jailbreak attack detection method according to claim 1 is characterized in that: The text data is converted into the vector representation through the embedding layer, and the vector representation includes a vector sequence, each vector corresponds to an element in the text data, and the conversion formula is: x = Embedding ( v )； in, x represents the vector representation, Embedding (·) indicates the embedding layer; v Represents input text data.

3. The large model jailbreak attack detection method according to claim 1, characterized in that: It also includes a training process of the gating network, a loss function is calculated based on the probability that the expert network is selected to process the vector representation based on the true probability and the prediction of the gating network, and the parameters of the gating network are adjusted by minimizing the loss function; the loss function expression is: ； in, L represents the loss function; K represents the number of expert networks; Indicates i The true probability of an expert network being selected; Indicates selection i An expert network processes the probability represented by the vector; x Represents vector representation.

4. The large model jailbreak attack detection method according to claim 1 is characterized in that: In the trained jailbreak attack detection model, the input of the first layer of MoE-Transformer structure is the vector representation, the input of the remaining layers of MoE-Transformer structure is the output of the previous layer, and the output of the last layer of MoE-Transformer structure is the high-level feature vector; the high-level feature vector outputs the jailbreak attack probability through the fully connected layer, and the calculation formula of the fully connected layer is: ； in, P Indicates the probability of jailbreak attack; Represents the sigmoid activation function; represents the weight of the fully connected layer; represents the bias of the fully connected layer; Represents a high-level feature vector.

5. The large model jailbreak attack detection method according to claim 1, characterized in that: Determining the jailbreak attack situation based on the jailbreak attack probability includes: The jailbreak attack probability is compared with a preset threshold value. If the jailbreak attack probability is greater than the preset threshold value, the text data is determined to be a jailbreak attack and a security alarm is triggered; otherwise, the text data is determined to be normal data.

6. The large model jailbreak attack detection method according to claim 1, characterized in that: The process of training the jailbreak attack detection model using the unlabeled learning method includes: Set the learning rate, batch size, and number of iterations; During training, the positive unlabeled learning method regards known attack samples as positive samples, and regards multiple unlabeled samples as unknown state samples; The jailbreak attack detection model predicts each unknown state sample. t The predicted probability of the unknown state sample exceeds the second preset threshold, then the first t The unknown state samples are regarded as positive samples; otherwise, the unlabeled state is maintained; Based on the jailbreak attack detection model, the focal loss is calculated for each positive sample based on its predicted probability; The Adam optimizer is used to dynamically adjust the parameters of the jailbreak attack detection model based on the focal loss.

7. The large model jailbreak attack detection method according to claim 6 is characterized in that: The expression of focal loss is: ； in, indicates focal loss; represents the weight coefficient; represents the focusing parameter; Indicates the jailbreak attack detection model for t The predicted probability of a positive sample.

8. The large model jailbreak attack detection method according to claim 7, characterized in that: The parameter adjustment formula is: ； in, Represents the parameters of the updated jailbreak attack detection model; Indicates the parameters of the jailbreak attack detection model before the update; represents the learning rate; Indicates focus loss gradient.

Images