Method and device for detecting and relieving jailbreak attack based on multi-level embedding analysis
By combining multi-level embedding analysis and diffusion models, the problems of coarse granularity and poor mitigation effect in the detection of jailbreak attacks using large language models are solved, achieving accurate identification and effective mitigation of jailbreak attacks, and improving the security robustness and adaptability of the model.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHEJIANG GONGSHANG UNIVERSITY
- Filing Date
- 2026-04-14
- Publication Date
- 2026-05-12
AI Technical Summary
Existing methods for detecting jailbreak attacks using large language models suffer from problems such as coarse detection granularity, insufficient modeling of internal representations, and poor mitigation effects. They struggle to identify deep jailbreak hints and adapt to evolving attack strategies, and lack effective semantic intervention and controllable reconstruction capabilities.
A multi-level embedding analysis method is adopted. By calculating the difference between benign and harmful embeddings in each layer, the anchor vector is extracted using singular value decomposition, and a diffusion model is constructed to detect and mitigate jailbreak attacks. This includes dynamic key layer localization, anchor vector updating, and diffusion model denoising processing of U-Net variant architecture.
It achieves accurate identification and effective mitigation of jailbreak attacks, improves the robustness and adaptability of detection, ensures the compliance and security of model output, avoids semantic drift or breakage issues, and is applicable to various large language model architectures.
Smart Images
Figure CN122020144A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence security technology, specifically to a method and apparatus for detecting and mitigating jailbreak attacks based on multi-level embedding analysis. Background Technology
[0002] In recent years, with the widespread application of large language models in scenarios such as dialogue generation, coding, and content creation, jailbreaking attacks targeting these models have gradually become a significant threat in the field of artificial intelligence security. Jailbreaking attacks refer to attackers using carefully designed input prompts to induce models to bypass existing alignment strategies and security boundaries, generating harmful, sensitive, or policy-violating responses. These attacks are often disguised as ordinary interactive commands, possessing strong stealth and misleading capabilities, greatly increasing the difficulty of detection, and potentially leading to serious ethical and security consequences.
[0003] Jailbreak attacks refer to attackers cleverly crafting prompts to induce models to bypass their built-in security rules or alignment mechanisms, outputting previously restricted illegal or sensitive content, such as violent threats, racial discrimination, inappropriate instructions, or privacy breaches. These attacks typically do not rely on access to model parameters; they can trigger high-risk outputs solely based on input statements. They are characterized by strong black-box nature, flexibility, and high detection difficulty, posing a substantial threat to the widespread application of large language models. To address this challenge, existing research has proposed various solutions at both the detection and defense levels, such as keyword filtering, rejection templates, security fine-tuning, and adversarial training, attempting to suppress jailbreak behavior from the input control or behavioral norms level. However, given the increasing complexity of jailbreak attacks and the continuous evolution of prompt design, existing technological systems still face many challenges.
[0004] Current jailbreak detection methods have the following shortcomings:
[0005] 1) Coarse modeling granularity makes it difficult to identify deep jailbreak hints: Most jailbreak detection methods use shallow strategies based on vocabulary or pattern matching, which cannot identify the illegal intent hidden in the hints through logical nesting, semantic ambiguity, role-playing, etc., resulting in a large number of evasion hints being missed.
[0006] 2) Ignoring differences in internal model representation leads to poor detection robustness: Current detection methods are mostly based on input text or output response features, lacking in-depth modeling of intermediate representations (such as embedding layers and multi-layer attention) during the processing process, making it difficult to capture the systematic deviations in the representation path between benign and jailbroken inputs.
[0007] 3) Poor generalization ability and difficulty in adapting to the evolution of attack strategies: The triggering modes of jailbreak attacks are highly diverse and rapidly evolving. Existing detection models often rely on static data training and lack dynamic adaptation mechanisms, resulting in a significant decrease in their ability to identify new attack samples.
[0008] At the same time, there are also some problems with the jailbreak mitigation part:
[0009] 1) Lack of semantic intervention mechanism: Most defense methods cannot effectively intervene in the intermediate representation of the model, so even if jailbreak behavior is detected, it cannot be prevented from continuing to propagate in the generation path.
[0010] 2) Lack of controllable restoration capability: Once the model response deviates, there is a lack of an "error correction-regeneration" mechanism; even if adversarial training or interpolation intervention is introduced, the scope of its influence is unclear, which can easily cause semantic drift or semantic breakage. Summary of the Invention
[0011] The purpose of this invention is to address the limitations of existing methods for detecting and defending against jailbreak attacks on large language models, particularly in terms of coarse detection granularity, insufficient model internal representation modeling, and poor mitigation effectiveness. This invention proposes a method and system for detecting and mitigating jailbreak attacks on large language models based on multi-level embedding analysis. This method aims to achieve a general-purpose security protection scheme for large language models with strong deep threat identification capabilities, accurate model representation modeling, and good defense performance, thereby improving the security robustness of mainstream large language models.
[0012] The objective of this invention is achieved through the following technical solution: a jailbreak attack detection and mitigation method based on multi-level embedding analysis, the method comprising:
[0013] S1. Calculate the benign average embedding and harmful average embedding of each layer of the large language model, and calculate the difference between the average embedding and the L2 norm of the input to determine the harmful key layer and the jailbreak key layer.
[0014] S2. Calculate the difference matrix between benign and harmful embeddings in the harmful key layer and the jailbreak key layer, and then use singular value decomposition to extract principal components to obtain the harmful anchor vector and the jailbreak anchor vector.
[0015] S3. Construct the difference matrix between the last token embedding and the benign average embedding of the input to the harmful key layer, and construct the difference matrix between the last token embedding and the harmful average embedding of the input to the jailbreak key layer. Perform singular value decomposition on the two difference matrices to obtain the harmful principal components and the jailbreak principal components. Calculate the matching degree between the principal component vectors and the anchor vectors based on cosine similarity to determine whether the input sequence constitutes a jailbreak attack. If it is determined to be a jailbreak attack, update the anchor vectors.
[0016] S4. Construct a diffusion model to mitigate jailbreak attacks. Build a dataset to train the diffusion model. Use the trained model to denoise the jailbreak key layer embeddings identified as jailbreak attacks as T-step noise, generating samples that approximate the real data distribution to achieve jailbreak mitigation.
[0017] Further, S1 includes:
[0018] The benign average embedding is calculated by: calculating the last token embedding of each benign input sample in layer l and averaging them;
[0019] The harmful average embedding is calculated by: calculating the last token embedding of each harmful input sample in layer l and averaging it;
[0020] The layers with the largest L2 norm between the input and the benign average embedding, which is greater than a preset threshold, are selected as the harmful key layers. The layers with the largest L2 norm between the input and the harmful average embedding, which is greater than a preset threshold, are selected as the jailbreak key layers.
[0021] Furthermore, the harmful anchor vector is constructed by creating a difference matrix between benign and harmful embeddings at the harmful key layer, and then using singular value decomposition to select the first column of the right singular vector as the harmful anchor vector;
[0022] The jailbreak anchor vector is obtained by constructing a difference matrix between jailbreak embedding and harmful embedding in the jailbreak key layer and then obtaining the jailbreak anchor vector through singular value decomposition.
[0023] The harmful principal components are obtained by performing singular value decomposition on the difference matrix between the embedding of the last token in the harmful key layer and the average embedding of the benign input in the harmful key layer.
[0024] The jailbreak principal components are obtained by performing singular value decomposition on the difference matrix between the embedding of the last token in the jailbreak key layer and the average embedding of the harmful input in the jailbreak key layer.
[0025] Furthermore, the specific steps for determining whether the input sequence constitutes a jailbreak attack are as follows:
[0026] The cosine similarity between the harmful principal component vector and the harmful anchor vector is calculated as the first judgment parameter; the cosine similarity between the jailbreak principal component vector and the jailbreak anchor vector is calculated as the second judgment parameter; when the first judgment parameter is greater than the sensitivity and the second judgment parameter is greater than the specificity; the sensitivity and specificity are the accuracy rates of the target samples that should be detected and the non-target samples that should not be detected, respectively, and are determined by the Youden index.
[0027] Furthermore, the diffusion model for mitigating jailbreak attacks includes:
[0028] The U-Net variant architecture is used as the backbone of the diffusion model, which includes an encoder, a bottleneck layer and a decoder. Temporal embedding conditions are introduced in each layer, and semantic fidelity is enhanced by residual connections. The U-Net variant architecture replaces the convolutional layers of the traditional U-Net with fully connected layers, introduces temporal embeddings in each layer of the network, and then adds skip connections between the corresponding layers of the encoder and decoder.
[0029] Furthermore, the time step embedding is as follows: a time step is randomly selected from a preset time step range, and a corresponding time step embedding vector is constructed through sine or cosine encoding. Specifically, when the dimension index of the embedding vector is even, sine time step encoding is selected, and when the dimension index of the embedding vector is odd, cosine time step encoding is selected.
[0030] Furthermore, the dataset used in the training process of the diffusion model is constructed as follows:
[0031] Based on benign input, harmful input, and jailbreak input in the training sample set;
[0032] The last token of the benign input passed through layer 0 is used as the benign embedding in step 0.
[0033] The jailbreak input is used as the last token passed through the l-th layer as the jailbreak embedding in the t-th step.
[0034] The current layer is treated as a time step to construct a training set including the benign embedding at step 0, the jailbreak embedding at step t, and the time step.
[0035] Furthermore, the specific steps for denoising the jailbreak key layer embedded as T-step noise, as determined by a jailbreak attack, are as follows:
[0036] After detecting a jailbreak attack, the key jailbreak layer j is located and its corresponding embedding is performed. Treated as T-step noise, according to the formula Denoising is performed, among which It is the intensity of the noise at each step. =1- , , This represents the noise predicted by the diffusion model. Random Gaussian noise is used to ensure sample diversity; high-quality samples that closely approximate the real data distribution are gradually generated from random noise using an iterative formula.
[0037] According to another aspect of the specification, this application also provides a jailbreak attack detection and mitigation device based on multi-level embedding analysis, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements the jailbreak attack detection and mitigation method based on multi-level embedding analysis.
[0038] According to another aspect of the specification, this application also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the aforementioned jailbreak attack detection and mitigation method based on multi-level embedding analysis.
[0039] The beneficial effects of this invention are:
[0040] 1. Improved Detection Accuracy. Traditional jailbreak attack detection methods rely on shallow feature matching, which struggles to identify deep semantic anomalies. This invention utilizes dynamic key layer localization and multi-level embedding analysis to calculate the L2 norm difference between embedding layers and extract principal components through singular value decomposition. This accurately identifies and mitigates jailbreak attacks, overcoming the limitations of traditional shallow detection methods. It can precisely capture abnormal patterns in the model's internal representation, significantly improving the detection accuracy for complex jailbreak attacks.
[0041] 2. Efficient Mitigation of Jailbreak Attacks. Traditional mitigation methods struggle to effectively intervene in the intermediate representations of the model, easily leading to semantic drift. This invention utilizes a diffusion model to perform semantically faithful denoising on jailbreak embeddings. A U-Net variant architecture diffusion model learns the mapping relationship between anomalous embeddings and secure embeddings. When an attack is detected, the model predicts the noise perturbation in the anomalous layer and performs denoising, restoring the jailbreak embedding to a benign semantic representation. This effectively mitigates jailbreak attacks, ensuring the compliance and security of the model output while avoiding semantic drift or fragmentation issues.
[0042] 3. Strong dynamic adaptability. Existing detection systems struggle to adapt to new attack strategies. This invention dynamically updates anchor vectors, incorporating recent jailbreak attacks into them. This continuously optimizes the attack detection benchmark, enhances the ability to detect new jailbreak attack patterns, and continuously improves detection capabilities, ensuring the long-term effectiveness of the defense system and enabling it to cope with constantly evolving attack methods.
[0043] 4. Good universality. Existing methods are often designed for specific model architectures, lacking universality and difficult to adapt to different large language models. The multi-level embedding analysis method of this invention does not depend on the parameter settings or architectural characteristics of a specific model. Its dynamic key layer localization mechanism can automatically adapt to the internal representation characteristics of different models, and the anchor vector generation and update strategy also has good model independence. Therefore, it can be applied to a variety of mainstream large language model architectures. Attached Figure Description
[0044] Figure 1 A schematic diagram of a jailbreak attack detection and mitigation method based on multi-level embedding analysis provided in an embodiment of the present invention;
[0045] Figure 2 An example architecture diagram provided for an embodiment of the present invention;
[0046] Figure 3 A schematic diagram illustrating the evaluation results of jailbreak attack detection performance provided in an embodiment of the present invention;
[0047] Figure 4 This is a schematic diagram of a jailbreak attack detection and mitigation device based on multi-level embedding analysis, provided as an embodiment of the present invention. Detailed Implementation
[0048] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.
[0049] like Figure 1 As shown, the present invention provides a jailbreak attack detection and mitigation method based on multi-level embedding analysis, the method steps of which are as follows:
[0050] S1: Multi-level Embedded Computation and Key Layer Localization
[0051] S2: Anchor Vector Generation
[0052] S3: Jailbreak Input Detection and Anchor Vector Update
[0053] S4: Jailbreak mitigation driven by a diffusion model
[0054] Corresponding framework such as Figure 1 As shown. The following is a detailed analysis of the function of each part using a specific embodiment:
[0055] S1: In this invention, the benign data set is derived from the Alpaca dataset, which contains a rich variety of real-world scenarios. A total of 850 benign prompts were randomly selected to form the benign segment of our dataset. For harmful input, the 520 suggestions from the AdvBench dataset and the 330 suggestions from the Hex-PH dataset are merged to form harmful segments. The jailbreak hints were generated by applying nine different jailbreak attacks to five different LLMs. From these hints, 850 were randomly selected as the jailbreak segments in the dataset. .
[0056] Next, we calculate the average embedding of the l-th layer of the benign input large language model: , where n represents the number of benign input samples, and the average embedding of the l-th layer of harmful input is calculated: For the location of the harmful layer: ,in This represents the embedding of the last token of input x in layer l. Here, the layer with the largest L2 norm between the input concept and the benign concept, which is greater than a certain threshold, is selected as the harmful layer, denoted as layer t. Similarly, after... We can then denote the jailbreak layer as layer j. In this way, we obtain the harmful critical layer t and the jailbreak critical layer j.
[0057] In this application, the text sequence is represented by a continuous vectorized representation obtained from each layer of the model to capture semantic and syntactic information; in this invention, the embedding is used as the basic data for generating the difference matrix and anchor vector to characterize the internal differences of benign, toxic and jailbroken inputs.
[0058] S2: First, construct the difference matrix between benign embeddings and harmful performance embeddings at the toxicity key layer t. ,in Indicates harmful layer embedding. It is the i-th harmful sample. Let be the i-th benign sample, and n represent the number of samples. Then, singular value decomposition is used to extract principal components, and the first column of the right singular vector is selected as the toxicity anchor vector. Similarly, a difference matrix between jailbreak embeddings and toxic embeddings can be constructed at the jailbreak-critical layer j. Next, singular value decomposition is performed to obtain the jailbreak anchor vector. .
[0059] Anchor vectors are directional vectors composed of principal components of the difference matrix, used to characterize typical embedding features of toxic or jailbreak inputs. In this invention, the cosine similarity between the anchor vector and the principal components of the input to be tested is used to determine jailbreak attacks. Furthermore, newly detected jailbreak embedding increments are fused into existing anchor vectors to continuously correct the detection benchmark.
[0060] S3: In this process, we can utilize the average embedding of the benign input in S1 at layer t. and the average embedding of harmful inputs in layer j Next, we only need to construct the corresponding difference matrix. and ,in This refers to the embedding of input x in the last token of layer t. This refers to the embedding of the last token of input x in layer j. Because the anchor vectors mentioned above are constructed from n data points in the dataset, they have good generalization ability; here, the difference matrix is constructed specifically for input x. Next, we will... and Perform singular value decomposition to obtain the corresponding principal components. and .
[0061] To determine whether an input sequence x constitutes a jailbreak attack, the degree of matching between its principal component vectors and anchor vectors needs to be calculated based on cosine similarity. Specifically, cosine similarity can be calculated using the following formula: and Then, a jailbreak check was performed: Sensitivity Specificity refers to the accuracy of detecting the target samples. The accuracy rate for detecting non-target samples that should not be detected is then used to determine the threshold using the Youden index. , .
[0062] After identifying a jailbreak attack, the anchor vector needs to be updated. The update expression for the anchor vector of the toxicity concept is as follows: , This is the learning rate for anchor vector updates. For the jailbreak concept, the anchor vector update is expressed as: , This is the learning rate for updating the anchor vector. After the update is complete, if the key layer is also located in the same layer for the next input x, the updated anchor vector can be used directly. and .
[0063] S4: This step aims to mitigate input embedding anomalies caused by jailbreak attacks. By training a diffusion model, the anomalous embeddings are restored to safe, normal embeddings, achieving semantic correction of the input and stable control of the model's output behavior. The model uses a variant of the U-Net architecture as the backbone of the diffusion model, including an encoder (downsampling block), a bottleneck layer, and a decoder (upsampling block). Each layer introduces temporal embedding conditions, and semantic fidelity is enhanced through residual connections. First, based on the training sample set... ,in This indicates positive input. This indicates harmful input. The jailbreak input is used to construct training samples as follows:
[0064] 1. The embedding of the last token after layer 0 is used here. Construct n .
[0065] 2. The embedding of the last token after the l-th layer is used here. Construct n×(L-1) .
[0066] 3. Treat the current layer number l as the time step t and add it to the training set.
[0067] The final dataset D*( , ,t) is used as the training set for the diffusion model.
[0068] The construction of the U-net variant involves replacing the convolutional layers of the traditional U-Net with fully connected layers, then introducing time-step embeddings into each network layer to enable the model to learn the noise distribution at different diffusion stages. Next, skip connections are added between the corresponding layers of the encoder and decoder to directly pass the original features to enhance semantic fidelity.
[0069] The addition of embedded noise can be represented as ,in, It is a benign embedding at step 0. This is step t of jailbreak embedding. It is Gaussian noise. It is the cumulative signal retention rate. This represents the proportion of the original signal energy retained at step t. It is the noise intensity injected in this step, and will This serves as the true label for the noise. The next step is to train the diffusion model (DM) to generate the predicted noise, calculate the loss, and then optimize the diffusion model.
[0070] The construction of time-step embeddings involves randomly selecting a time step from a preset time-step range t∈(0,L) and constructing a corresponding time-step embedding vector using sine / cosine time coding, which serves as one of the input conditions for the diffusion model. Specifically, this is achieved through sine / cosine coding, which can be expressed by the formula... Where t is the time step of the diffusion process, d is the total dimension of the temporal embedding vector, and i is the dimension index of the embedding vector. This allows us to convert the time step into the desired temporal embedding.
[0071] The jailbreak mitigation part is essentially the denoising process of the diffusion model. After detecting a jailbreak attack, the key jailbreak layer j is located and its corresponding embedding is performed. Treating it as T-step noise, the next step is to follow the formula. Denoising is performed, among which It is the intensity of the noise at each step. =1- , , This represents the noise predicted by the diffusion model. Random Gaussian noise is used to ensure sample diversity. By iterating this formula (t=T~t=1), the model can gradually generate high-quality samples that closely approximate the real data distribution from random noise, thus achieving jailbreak mitigation.
[0072] Figure 2This is an architecture diagram of an example of the present invention, including a large language model and a jailbreak detection module and a jailbreak mitigation module connected thereto. The large language model sequentially comprises a multi-layer structure from hidden layer 1 to hidden layer L. A jailbreak detection module is connected at the located jailbreak-critical layer (represented by hidden layer x in the diagram) to determine if the embedding representation of that layer is a jailbreak attack. When the determination result is a jailbreak attack, the detection result is processed by the jailbreak mitigation module to denoise and restore the embedding of that layer, generating a secure embedding which is then passed to the subsequent hidden layer (represented by hidden layer x+1 in the diagram). When the determination result is a non-jailbreak attack, the embedding is directly passed to the subsequent hidden layer for further processing, ultimately generating a benign output. After undergoing the above detection and mitigation process, the harmful input can effectively reduce the risk of jailbreak attacks and ensure the compliance and security of the output.
[0073] The jailbreak mitigation module is the model trained in S4 above. It is implemented based on the diffusion model of the U-Net variant architecture. It denoises the jailbreak key layer embeddings that are detected as jailbreak attacks as T-step noise input. By predicting and removing abnormal perturbations, it restores the jailbreak embeddings to an embedding representation that is close to the benign distribution. Thus, it effectively eliminates the harmful effects of jailbreak attacks without destroying the original semantic coherence.
[0074] Figure 3 This is a schematic diagram illustrating the evaluation results of jailbreak attack detection performance in an embodiment of the present invention. The horizontal axis represents the method name of different jailbreak attacks, and the vertical axis represents accuracy and F1 score, respectively. This figure shows the detection performance scores of the jailbreak detection method based on multi-level embedding analysis proposed in this invention under various jailbreak attack types. As can be seen from the figure, the method of the present invention has achieved good results on various attack methods, indicating that the method can achieve stable and efficient detection performance in diverse jailbreak attack scenarios.
[0075] To clearly demonstrate a method for detecting and mitigating jailbreak attacks on large language models based on multi-level embedding analysis, the embodiment also provides an apparatus for such a method, including a memory and a processor. The memory stores a computer program, and the processor, when executing the computer program, implements the method for detecting and mitigating jailbreak attacks on large language models based on multi-level embedding analysis.
[0076] Corresponding to the aforementioned embodiment of a jailbreak attack detection and mitigation method based on multi-level embedding analysis, the present invention also provides an embodiment of a jailbreak attack detection and mitigation device based on multi-level embedding analysis.
[0077] See Figure 4The present invention provides a jailbreak attack detection and mitigation device based on multi-level embedding analysis, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement a jailbreak attack detection and mitigation method based on multi-level embedding analysis in the above embodiment.
[0078] The embodiment of the jailbreak attack detection and mitigation device based on multi-level embedded analysis provided by this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 4 The diagram shown is a hardware structure diagram of any data processing-capable device, where the jailbreak attack detection and mitigation device based on multi-level embedded analysis provided by this invention is located. (Except for...) Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.
[0079] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.
[0080] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0081] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements a jailbreak attack detection and mitigation method based on multi-level embedding analysis as described in the above embodiments.
[0082] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.
[0083] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the jailbreak attack detection and mitigation method based on multi-level embedding analysis.
[0084] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.
[0085] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. This application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.
Claims
1. A jailbreak attack detection and mitigation method based on multi-level embedding analysis, characterized in that, The method includes: S1. Calculate the benign average embedding and harmful average embedding of each layer of the large language model, and calculate the difference between the average embedding and the L2 norm of the input to determine the harmful key layer and the jailbreak key layer. S2. Calculate the difference matrix between benign and harmful embeddings in the harmful key layer and the jailbreak key layer, and then use singular value decomposition to extract principal components to obtain the harmful anchor vector and the jailbreak anchor vector. S3. Construct the difference matrix between the last token embedding and the benign average embedding of the input to the harmful key layer, and construct the difference matrix between the last token embedding and the harmful average embedding of the input to the jailbreak key layer. Perform singular value decomposition on the two difference matrices to obtain the harmful principal components and the jailbreak principal components. Calculate the matching degree between the principal component vectors and the anchor vectors based on cosine similarity to determine whether the input sequence constitutes a jailbreak attack. If it is determined to be a jailbreak attack, update the anchor vectors. S4. Construct a diffusion model to mitigate jailbreak attacks. Build a dataset to train the diffusion model. Use the trained model to denoise the jailbreak key layer embeddings identified as jailbreak attacks as T-step noise, generating samples that approximate the real data distribution to achieve jailbreak mitigation.
2. The jailbreak attack detection and mitigation method based on multi-level embedding analysis according to claim 1, characterized in that, S1 includes: The benign average embedding is calculated by: calculating the last token embedding of each benign input sample in layer l and averaging them; The harmful average embedding is calculated by: calculating the last token embedding of each harmful input sample in layer l and averaging it; The layers with the largest L2 norm between the input and the benign average embedding, which is greater than a preset threshold, are selected as the harmful key layers. The layers with the largest L2 norm between the input and the harmful average embedding, which is greater than a preset threshold, are selected as the jailbreak key layers.
3. The jailbreak attack detection and mitigation method based on multi-level embedding analysis according to claim 1, characterized in that, The harmful anchor vector is constructed by building a difference matrix between benign and harmful embeddings at the harmful key layer, and then using singular value decomposition to select the first column of the right singular vector as the harmful anchor vector. The jailbreak anchor vector is obtained by constructing a difference matrix between jailbreak embedding and harmful embedding in the jailbreak key layer and then obtaining the jailbreak anchor vector through singular value decomposition. The harmful principal components are obtained by performing singular value decomposition on the difference matrix between the embedding of the last token in the harmful key layer and the average embedding of the benign input in the harmful key layer. The jailbreak principal components are obtained by performing singular value decomposition on the difference matrix between the embedding of the last token in the jailbreak key layer and the average embedding of the harmful input in the jailbreak key layer.
4. The jailbreak attack detection and mitigation method based on multi-level embedding analysis according to claim 1, characterized in that, The specific steps for determining whether the input sequence constitutes a jailbreak attack are as follows: The cosine similarity between the harmful principal component vector and the harmful anchor vector is calculated as the first judgment parameter; the cosine similarity between the jailbreak principal component vector and the jailbreak anchor vector is calculated as the second judgment parameter; when the first judgment parameter is greater than the sensitivity and the second judgment parameter is greater than the specificity; the sensitivity and specificity are the accuracy rates of the target samples that should be detected and the non-target samples that should not be detected, respectively, and are determined by the Youden index.
5. The jailbreak attack detection and mitigation method based on multi-level embedding analysis according to claim 1, characterized in that, The diffusion model for mitigating jailbreak attacks includes: The U-Net variant architecture is used as the backbone of the diffusion model, which includes an encoder, a bottleneck layer and a decoder. Temporal embedding conditions are introduced in each layer, and semantic fidelity is enhanced by residual connections. The U-Net variant architecture replaces the convolutional layers of the traditional U-Net with fully connected layers, introduces temporal embeddings in each layer of the network, and then adds skip connections between the corresponding layers of the encoder and decoder.
6. The jailbreak attack detection and mitigation method based on multi-level embedding analysis according to claim 5, characterized in that, The time step embedding is as follows: a time step is randomly selected from a preset time step range, and a corresponding time step embedding vector is constructed through sine or cosine encoding. Specifically, when the dimension index of the embedding vector is even, sine time step encoding is selected, and when the dimension index of the embedding vector is odd, cosine time step encoding is selected.
7. The jailbreak attack detection and mitigation method based on multi-level embedding analysis according to claim 1, characterized in that, The dataset for training the diffusion model is constructed as follows: Based on benign input, harmful input, and jailbreak input in the training sample set; The last token of the benign input passed through layer 0 is used as the benign embedding in step 0. The jailbreak input is used as the last token passed through the l-th layer as the jailbreak embedding in the t-th step. The current layer is treated as a time step to construct a training set including the benign embedding at step 0, the jailbreak embedding at step t, and the time step.
8. The jailbreak attack detection and mitigation method based on multi-level embedding analysis according to claim 1, characterized in that, The specific steps for denoising the jailbreak key layer identified as a jailbreak attack as T-step noise are as follows: After detecting a jailbreak attack, the jailbreak key layer j is located, and its corresponding embedding is regarded as T-step noise. The noise is denoised iteratively based on the noise predicted by the model, and a random noise term is added in each iteration. High-quality samples that closely approximate the real data distribution are gradually generated from random noise using an iterative formula.
9. A jailbreak attack detection and mitigation device based on multi-level embedding analysis, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements a jailbreak attack detection and mitigation method based on multi-level embedding analysis as described in any one of claims 1-8.
10. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements a jailbreak attack detection and mitigation method based on multi-level embedding analysis as described in any one of claims 1-8.