A large model jailbreaking attack detection method and device based on hidden layer state
By locating the security-critical layer in a large language model and extracting security vectors using principal component analysis, an efficient jailbreak attack detection method is constructed. This method solves the problems of insufficient detection accuracy, poor real-time performance, and high inference overhead in existing technologies, achieving high-precision, low-overhead, and real-time jailbreak attack detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- UNIV OF SCI & TECH OF CHINA
- Filing Date
- 2025-12-17
- Publication Date
- 2026-07-31
AI Technical Summary
Existing methods for detecting jailbreak attacks using large language models cannot simultaneously achieve high accuracy, real-time performance, and low overhead, resulting in problems such as insufficient detection accuracy, poor real-time performance, and high inference overhead.
By extracting the hidden states of a large language model, locating the security-critical layer, extracting security vectors using principal component analysis, and identifying jailbreak attacks using a classifier, an efficient jailbreak attack detection scheme is constructed.
It achieves high-precision, low-overhead, and real-time jailbreak attack detection, improves the security of large language models, and can identify potential jailbreak attack behaviors at an early stage.
Smart Images
Figure CN121959552B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of artificial intelligence security technology, specifically relating to a method and apparatus for detecting large-scale jailbreak attacks based on hidden layer states. Background Technology
[0002] With the widespread application of large language models, their security faces serious challenges, especially jailbreak attacks. Jailbreak attacks refer to attackers designing deceptive inputs to induce the model to generate harmful content, such as violent speech or malicious code.
[0003] Existing jailbreak attack detection methods are mainly divided into three categories, but all of them have inherent flaws:
[0004] While input-side detection methods (such as keyword filtering and content moderation APIs) are simple to implement, they rely on fixed rules or external models, resulting in insufficient detection accuracy and difficulty in dealing with new types of attacks.
[0005] The output detection method reviews all content after the model has generated it. While this ensures the safety of the results, it suffers from a significant lag and cannot achieve real-time interception.
[0006] Model behavior analysis methods detect anomalies by monitoring dynamic features (such as perplexity) during the generation process, which has good robustness. However, it requires continuous computation of dynamic features, which significantly increases inference overhead and affects model efficiency.
[0007] In summary, existing technologies cannot simultaneously achieve high accuracy, real-time performance, and low overhead, necessitating a more efficient detection solution. Summary of the Invention
[0008] This application proposes a jailbreak attack detection scheme based on a large language model with hidden layer states, aiming to solve the problems of insufficient detection accuracy, poor real-time performance, and high inference overhead in existing technologies.
[0009] The first aspect of this application provides a method for detecting jailbreak attacks on large language models based on hidden state, including:
[0010] Extract the hidden state of the user input to be detected in the set of security critical layers, wherein the hidden state is the output of the intermediate layer of the large language model. The set of security critical layers is predetermined by constructing a jailbreak attack test set, applying random perturbation to the hidden state of each network layer of the large language model, evaluating the change in the success rate of jailbreak attacks before and after the perturbation, quantifying the importance of each network layer to security protection, and determining the set of security critical layers based on the importance.
[0011] The hidden state of each layer is projected onto the security vector corresponding to the layer, wherein the security vector is pre-extracted in the following way: for the set of security critical layers, the hidden state is extracted from the harmful instruction set and the harmless instruction set respectively, and the security vector of each security critical layer is extracted by principal component analysis.
[0012] All projected values are concatenated to construct a feature vector, which is then input into a pre-trained classifier. The classifier output is used to determine whether the user input to be detected constitutes a jailbreak attack.
[0013] In some embodiments of this application, the random perturbation is the addition of a noise vector conforming to a Gaussian distribution, wherein the noise vector is sampled from a normal distribution with zero mean and adjustable variance.
[0014] In some embodiments of this application, the random perturbation is expressed by a formula Implementation, including:
[0015] The hidden state generated by the large language model at the k-th network layer for the input instruction x. This refers to the hidden layer state after applying a perturbation to the network layer. The noise vector sampled from the Gaussian distribution. This is a scaling factor used to adjust the noise intensity.
[0016] In some embodiments of this application, the evaluation of the change in the success rate of jailbreak attacks before and after the perturbation, and the quantification of the importance of each network layer to security protection, include:
[0017] Using formula Calculate the importance score of the network layer, where,
[0018] For importance scores, The jailbreak success rate before performing perturbation operations on the large language model. The jailbreak success rate after performing a perturbation operation on the large language model is calculated. The scaling factor is mentioned.
[0019] In some embodiments of this application, determining the set of security critical layers based on the importance includes:
[0020] Network layers whose importance scores exceed a preset threshold are selected to form the set of security-critical layers; or
[0021] A predetermined number of network layers with the highest importance scores are selected to form the set of security-critical layers.
[0022] In some embodiments of this application, the extraction of the security vector for each security-critical layer via principal component analysis includes:
[0023] Calculate the average hidden state activated by harmful and harmless instruction sets in the security-critical layer;
[0024] The average hidden state is used to decentralize all hidden states extracted from the harmful and harmless instruction sets;
[0025] The covariance matrix is calculated based on the decentralized hidden layer state;
[0026] The covariance matrix is decomposed into eigenvalues, and the eigenvector corresponding to the largest eigenvalue is selected as the safety vector.
[0027] In some embodiments of this application, the projection value is obtained through... calculate,
[0028] in, This refers to the hidden layer state. This refers to the average hidden layer state. This is the security vector.
[0029] In some embodiments of this application, determining whether the user input to be detected is a jailbreak attack based on the classifier output includes:
[0030] If the predicted value output by the classifier exceeds a preset threshold, the user input to be detected is determined to be a jailbreak attack.
[0031] A second aspect of this application provides a jailbreak attack detection device based on hidden layer states for large language models, comprising:
[0032] An extraction module is used to extract the hidden state of the user input to be detected in the set of security critical layers, wherein the hidden state is the output of the intermediate layer of the large language model, and the set of security critical layers is predetermined by the following method: constructing a jailbreak attack test set, applying random perturbation to the hidden state of each network layer of the large language model, evaluating the change in the success rate of jailbreak attacks before and after the perturbation, quantifying the importance of each network layer to security protection, and determining the set of security critical layers based on the importance;
[0033] The projection module is used to project the hidden state of each layer onto the security vector corresponding to the layer. The security vector is pre-extracted in the following way: for the set of security critical layers, the hidden state is extracted from the harmful instruction set and the harmless instruction set respectively, and the security vector of each security critical layer is extracted by principal component analysis.
[0034] The detection module is used to concatenate all projection values to construct a feature vector, and input the feature vector into a pre-trained classifier. Based on the classifier output, it determines whether the user input to be detected is a jailbreak attack.
[0035] In summary, the large language model jailbreak attack detection method and apparatus based on hidden layer states provided in the embodiments of this application quantify layer importance through perturbation experiments to locate security-critical layers. Because this technology can accurately identify the network layers most sensitive to security protection, it improves the accuracy and targeting of detection, overcoming the deficiency of insufficient detection accuracy at the input end. Security vectors are extracted through PCA analysis; because dimensionality reduction processing can efficiently capture the essential differences between harmful and harmless semantics, it significantly reduces computational overhead and solves the high overhead problem of model behavior analysis. Jailbreak attacks are detected through real-time projection and classifier techniques; because hidden layer states are analyzed in real-time during model inference, early interception is achieved, eliminating the lag in output-end detection. Finally, the synergistic integration of the above technologies forms a high-precision, low-overhead, and real-time jailbreak attack detection scheme, comprehensively improving the security of large language models. Attached Figure Description
[0036] The features and advantages of this application will be more clearly understood by referring to the accompanying drawings, which are illustrative and should not be construed as limiting the application in any way. In the drawings:
[0037] Figure 1 This is a diagram illustrating the overall framework of the large-scale jailbreak attack detection method based on hidden layer states proposed in this application.
[0038] Figure 2 This is a schematic diagram of a computer system to which this application applies;
[0039] Figure 3 This is a flowchart illustrating a method for detecting jailbreak attacks based on hidden layer states of a large language model, according to some embodiments of this application.
[0040] Figure 4 This is a flowchart of the safety critical layer location method of this application;
[0041] Figure 5 This is a flowchart of the security vector extraction method of this application;
[0042] Figure 6 This is a flowchart of the online real-time jailbreak attack detection method of this application;
[0043] Figure 7 This is a schematic diagram of a jailbreak attack detection device based on a large language model with hidden layer states, according to some embodiments of this application. Detailed Implementation
[0044] In the following detailed description, numerous specific details of this application are illustrated by example to provide a thorough understanding of the relevant disclosure. However, it will be apparent to those skilled in the art that this application can be practiced without these details. It should be understood that the terms “system,” “apparatus,” “unit,” and / or “module” used in this application are one way of distinguishing different parts, elements, sections, or components at different levels in a sequential arrangement. However, these terms may be replaced with other expressions if other expressions can achieve the same purpose.
[0045] It should be understood that when a device, unit, or module is referred to as being "on," "connected to," or "coupled to" another device, unit, or module, it may be directly connected to or coupled to or communicate with other devices, units, or modules, or there may be intermediate devices, units, or modules present, unless the context explicitly indicates otherwise. For example, the term "and / or" as used herein includes any one and all combinations of one or more of the relevant listed items.
[0046] The terminology used in this application is for the purpose of describing specific embodiments only and is not intended to limit the scope of this application. As shown in the specification and claims of this application, unless the context clearly indicates otherwise, words such as "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate that explicitly identified features, integrals, steps, operations, elements, and / or components are included, and such expressions do not constitute an exclusive list, and other features, integrals, steps, operations, elements, and / or components may also be included.
[0047] Referring to the following description and accompanying drawings, these and other features and characteristics, operating methods, functions of related structural elements, combinations of parts, and economics of manufacture of this application can be better understood, wherein the description and drawings form part of the specification. However, it is clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. It is understood that the drawings are not drawn to scale.
[0048] Various structural diagrams are used in this application to illustrate various variations of the embodiments according to this application. It should be understood that the preceding or following structures are not intended to limit this application. The scope of protection of this application is determined by the claims.
[0049] Large language models have developed rapidly in recent years, demonstrating enormous potential in numerous industries such as education, healthcare, finance, and scientific research. However, their security issues are becoming increasingly prominent. Jailbreak attacks are a prime example of this security threat. Attackers can bypass the model's internal security mechanisms simply by designing deceptive text inputs, inducing the model to generate hate speech, commit illegal or criminal activities, leak sensitive information, or even output malicious code and other harmful content. This attack method poses a significant threat to personal privacy and social security, thus necessitating the adoption of security protection technologies to effectively identify jailbreak attacks and enhance the robustness of large language models.
[0050] To address this security threat, existing jailbreak attack detection methods can be categorized into three types: input detection, output detection, and model behavior analysis. The first type of method reviews the user-input text, typically relying on keyword matching or calling content moderation APIs to determine if the input poses a security risk. While these methods are simple to implement and have low overhead, their detection accuracy is insufficient. The second type of method performs a secondary review of the model's generated results to ensure the security of the generated content. This method has a significant lag, requiring all text to be generated before detecting whether it is harmful, thus preventing real-time output. The third type of method analyzes the model's dynamic behavioral characteristics (such as the perplexity of the generated content and changes in security-critical gradients) in real time during inference to determine if the model's behavior is abnormal. While this behavior-based statistical approach offers better robustness, it comes at the cost of significantly increasing the model's inference overhead.
[0051] In summary, existing technologies cannot simultaneously achieve high accuracy, real-time performance, and low overhead, necessitating a more efficient detection solution.
[0052] In view of this, this application proposes a jailbreak attack detection method based on the hidden state of the model. This method is based on the following technical discovery: when a large language model encounters "safe" and "insecure" commands, the activation patterns, attention distributions, and semantic feature representations within its neural network often exhibit fundamental differences. These differences are ultimately reflected in the changes of the model's hidden state, constituting a multidimensional representation of the model's hidden state. Based on this phenomenon, this application, by effectively capturing and modeling the differential features in the hidden state, can identify potential jailbreak attack behaviors before the model generates harmful content, thereby achieving proactive security defense.
[0053] Figure 1 This is a diagram illustrating the overall framework of the large-scale jailbreak attack detection method based on hidden state proposed in this application. Figure 1As shown, the entire jailbreak attack detection process consists of three parts: security-critical layer localization, security vector extraction, and jailbreak command detection. In security-critical layer localization, the system adds perturbations to the intermediate hidden state of the original model to affect the model's security, thereby identifying the layers with the greatest impact on security. Next is security vector extraction, which involves inputting harmful and harmless commands, and then using PCA to analyze the feature differences between the two commands to obtain a security vector that can effectively distinguish them. Finally, using the aforementioned security vector, a classifier is trained to classify user input.
[0054] Figure 2 This is a schematic diagram of a computer system to which this application applies. Figure 2 The computer system shown includes a security analysis server connected to a jailbreak attack database and a large language model database. Based on historical data from the jailbreak attack database, the security analysis server processes and judges user input to the large language model in real time, ultimately outputting a detection result indicating whether the user input constitutes a jailbreak attack. Wherein:
[0055] The jailbreak attack database is a data storage component in the computer system used to support the offline preparation and online detection processes of the security analysis server. Specifically, it includes:
[0056] Store a jailbreak attack test suite to evaluate changes in the success rate of jailbreak attacks using the model;
[0057] Stores harmful and harmless instruction sets for principal component analysis in secure vector extraction.
[0058] The jailbreak attack database can be any of a single-machine, cluster, or distributed database.
[0059] The security analysis server is the core processing unit of the computer system, responsible for executing the entire process of jailbreak attack detection. Specifically, it includes:
[0060] During the offline preparation phase, the security critical layer of the large language model is located and the security vector is extracted: the importance of each network layer to security protection is quantified through perturbation experiments, the security critical layer is determined, and the discriminative security vector is extracted based on the difference in hidden layer states between harmful and harmless instruction sets.
[0061] During the online detection phase, jailbreak attack identification is achieved through real-time projection and classifier judgment: the hidden state input by the user is projected onto the security vector to construct a feature vector, and the judgment result is output using a pre-trained classifier.
[0062] The security analysis server can be a standalone, clustered, or distributed server.
[0063] The large language model is the core computational component of the computer system, responsible for processing input instructions and generating hidden states. Specifically, it includes:
[0064] During the offline preparation phase, harmful and harmless instruction sets are processed to generate hidden layer states for safety-critical layer localization and safety vector extraction.
[0065] During the online detection phase, user input commands are processed to generate hidden layer states for real-time projection and classifier determination.
[0066] The large language model is a pre-trained model that can be deployed as a standalone service (such as a cloud API) or integrated with a security analysis server. When deployed independently, it supports different environments such as single-machine, cluster, or distributed servers.
[0067] Figure 3 This is a flowchart illustrating a method for detecting jailbreak attacks based on hidden state of a large language model, according to some embodiments of this application. In some embodiments, the method for detecting jailbreak attacks based on hidden state of a large language model is composed of... Figure 2 The security analysis server shown is executing.
[0068] The jailbreak attack detection method includes an offline preparation phase and an online detection phase. The offline preparation phase constructs the basic parameters required for detection through security critical layer localization and security vector extraction. The online detection phase, based on these basic parameters, uses real-time projection and a classifier to identify jailbreak attacks input by the user. The offline preparation phase includes:
[0069] Safety-critical layer positioning:
[0070] This application first needs to identify which network layers in the large language model play a decisive role in resisting jailbreak attacks and maintaining content security. This application defines these network layers as "security-critical layers".
[0071] The core of security-critical layer identification is to quantitatively assess the importance of a layer in the security system by applying controlled perturbations to the hidden state of a specific layer in the model and observing the impact of these perturbations on the model's processing of jailbreak attack commands. The more significant the impact on jailbreak success rate, the higher the importance of the layer in security protection. Figure 4 This is a flowchart of the safety critical layer location method. The detailed implementation steps are described below:
[0072] Step 1.1: Constructing a jailbreak attack test suite and security assessment method
[0073] First, we need to prepare a jailbreak attack dataset for evaluation, denoted as . This dataset contains a series of instructions designed to induce large language models to generate inappropriate, harmful, or security-violating content; for example, guiding models to generate content involving violence, fraud, or other illegal activities.
[0074] Subsequently, in the original large-scale language model (denoted as) without any intervention On the test set, the initial jailbreak attack success rate was evaluated. For each input instruction in the test set The model will generate an output. An evaluation function is needed. To determine the output Whether a jailbreak constitutes a successful jailbreak. This function can be a binary classification function based on keyword matching, content classifiers, or manual annotation. It returns 1 if the jailbreak is successful, and 0 otherwise. Initial jailbreak attack success rate. The calculation formula is as follows:
[0075]
[0076] This value This will serve as a baseline for subsequent perturbation experiments.
[0077] Step 1.2: Apply random perturbations to the hidden state of the model.
[0078] Large language models are usually composed of It consists of a series of sequentially connected Transformer layers. For an input , No. layer( The output of ) is the hidden state, denoted as .
[0079] The localization method in this application will sequentially traverse each or part of the selected network layers in the model. For the current layer to be tested... Layer, the hidden state of its output Inject a random noise vector and multiply by the scaling factor The hidden state after perturbation is obtained. :
[0080] ,
[0081] Wherein, noise vector Typically, samples are taken from a multivariate normal distribution (Gaussian distribution), i.e.
[0082] Scaling factor The value of needs to be adjusted experimentally to ensure that the perturbation can significantly affect the model's security capabilities without completely destroying the model's normal semantic processing capabilities.
[0083] This perturbation process can be viewed as creating a temporary perturbation that only occurs on the [number]th [day]. The "perturbation model" with modified layers When input go through At that time, before Layer Calculation and Original Model Similarly, in the first The layer's output is replaced with Then the perturbed state is fed into the subsequent... Layer to the first The layer continues its calculations.
[0084] Step 1.3: Evaluate the success rate of jailbreak attacks after perturbation
[0085] For each disturbed layer Using the same jailbreak attack test suite For the perturbation model An assessment was conducted to calculate the success rate of the jailbreak attack. :
[0086] ,
[0087] Step 1.4: Quantify the importance of each level and select the security-critical layer.
[0088] By comparing the success rates after perturbation Compared with the benchmark success rate , can quantify the first The importance of the layer. Importance score. Defined as:
[0089] ,
[0090] in Indicates the first The scaling factor used when applying perturbations to a layer. The larger the value, the better for the th... The more significant the impact of perturbations on the hidden layer states on the model's final security decision, the more critical information related to security judgments the state of that layer contains, and therefore it has a higher importance in the model's security protection system.
[0091] After calculating the importance scores of all layers to be tested Then, these scores are sorted. The scores with the highest importance are selected. These network layers constitute a "set of security-critical layers", denoted as [missing information]. . The value is a preset hyperparameter, for example, it can be selected as 10% or 20% of the total number of layers, or all layers whose scores exceed a certain threshold.
[0092] Through the above steps, this application can accurately and quantitatively identify the set of network layers that are most critical to the security performance of large language models, laying the foundation for subsequent security vector extraction and attack detection.
[0093] Security Vector Extraction:
[0094] After locating the security-critical layer, the "security vector" is extracted. The goal is to find a direction in the hidden space of this layer that maximizes the similarity between harmful instructions and the vector, and minimizes the similarity between harmless instructions and the vector. To this end, this application inputs a large number of "harmful instruction sets" and "harmless instruction sets" into the model, extracts their hidden states, and performs PCA on these data. The first-order principal component with the largest variance is taken as the "security vector," which points in the direction of change from harmless to harmful content representation.
[0095] Figure 5 This is a flowchart of the security vector extraction method in this application. For example... Figure 5 As shown, the security vector extraction method specifically includes:
[0096] Step 2.1: Construct a dataset of harmful and harmless instructions
[0097] To learn to distinguish between harmful and harmless intentions, two types of instruction datasets are needed:
[0098] Harmful instruction sets This dataset contains a clearly defined set of malicious issues or jailbreak attack instructions. The dataset can be reused from step 1.1. Or use a broader set of harmful content.
[0099] Harmless instruction set This dataset contains a series of normal, security-compliant routine questions or instructions. The size and domain distribution of this dataset should be consistent with... To ensure fairness in the comparison.
[0100] Step 2.2: Extract the hidden state of the safety-critical layer
[0101] For the set of safety critical layers Each layer in , respectively and All instructions are input into the original large language model. In the middle, and record the model in the first... Hidden state of a layer.
[0102] For harmful instruction sets The extracted set of hidden states is:
[0103] ,
[0104] For harmless instruction sets The extracted set of hidden states is:
[0105] ,
[0106] and They represent the first The distribution of the activation state of the layer when processing two different types of intention input.
[0107] Step 2.3: Calculate the "safety vector" based on principal component analysis (PCA)
[0108] The core idea of this application is that, at the critical layer In the high-dimensional activation space, the hidden states corresponding to harmful and harmless instructions form two distinct clusters. We employ Principal Component Analysis (PCA) to extract the most important direction distinguishing these two types of samples in this space, namely the "safety vector".
[0109] First, calculate the two instruction sets in the second... Average hidden state of layer activation (i.e., cluster center):
[0110] ,
[0111] Decentralize all hidden states:
[0112] ,
[0113] Perform PCA to determine the main direction:
[0114] (a) Calculate the covariance matrix: based on the centered dataset Calculate its covariance matrix
[0115] (b) Eigenvalue decomposition: for the covariance matrix Eigenvalue decomposition yields a set of eigenvectors and their corresponding eigenvalues.
[0116] (c) Selecting the first principal component: Select the eigenvector corresponding to the largest eigenvalue. This vector represents the direction of maximum variance in the dataset, i.e., the first principal component. .
[0117] Set of safety-critical layers Repeat the above steps for each layer to obtain a set of security vectors. Each Each is a unit vector with the same dimension as the hidden state of the model, capturing the first... The core direction of layer-based semantic differentiation between secure and insecure layers.
[0118] After constructing the basic parameters required for detection through offline processing, a real-time jailbreak attack detection system can be built. During model inference, this system determines whether the input contains malicious intent by analyzing the alignment between the hidden state and the security vector.
[0119] Figure 6 This is a flowchart of the online real-time jailbreak attack detection method described in this application. Figure 6 As shown, the method for detecting jailbreak attacks based on hidden layer states of a large language model includes:
[0120] S310, extract the hidden state of the user input to be detected in the set of security critical layers, wherein the hidden state is the output of the intermediate layer of the large language model, and the set of security critical layers is predetermined by the following method: construct a jailbreak attack test set, apply random perturbation to the hidden state of each network layer of the large language model, evaluate the change in the success rate of jailbreak attacks before and after the perturbation, quantify the importance of each network layer to security protection, and determine the set of security critical layers according to the importance.
[0121] First, for each new user input, forward inference of the model is performed to extract the hidden states of the safety-critical layers. These hidden states are the outputs of the intermediate layers of the large language model, and the set of safety-critical layers is pre-extracted based on the methods described in steps 1.1-1.4.
[0122] S320, the hidden state of each layer is projected onto the security vector corresponding to the layer, wherein the security vector is pre-extracted in the following way: for the set of security critical layers, the hidden state is extracted from the harmful instruction set and the harmless instruction set respectively, and the security vector of each security critical layer is extracted by principal component analysis.
[0123] Next, after obtaining the hidden state, the hidden state of each layer is projected onto the corresponding security vector to obtain the projection value. ,
[0124] in, It is a hidden state. This represents the average hidden state. For security vectors.
[0125] The security vector is extracted in advance from the differences in hidden state between harmful and harmless instruction sets based on the method described in steps 2.1-2.3 using principal component analysis (PCA).
[0126] S330, concatenate all projection values to construct a feature vector, and input the feature vector into a pre-trained classifier. Determine whether the user input to be detected is a jailbreak attack based on the classifier output.
[0127] Finally, project the values of all layers. Concatenate them sequentially to form a feature vector. This vector comprehensively reflects the security-related semantic information of user input at multiple levels, serving as a representation of user input.
[0128] The feature vector is then input into the trained classifier. Make a judgment. The classifier is a binary classifier trained using the logistic regression algorithm, and its output is represented as follows:
[0129] ,
[0130] in, and These are the parameters of the classifier. yes Activation function.
[0131] The system determines whether the user input constitutes a jailbreak attack based on the classifier's output: if the predicted value of the classifier's output exceeds a preset threshold... If the input is deemed harmful, it is considered harmful; otherwise, it is considered harmless.
[0132] The classifier is trained using the classic cross-entropy function.
[0133] ,
[0134] Here Does the instruction represent harm? This represents a harmful instruction. Represents harmless instructions
[0135] Therefore, this application constructs a complete and efficient large-scale language model jailbreak attack detection scheme through a three-step method of "location-extraction-detection". This scheme achieves accurate identification of malicious attacks without modifying model weights, utilizing only the model's internal state information, thus providing a guarantee for the content security of large-scale language models.
[0136] Figure 7 This is a schematic diagram of a jailbreak attack detection device based on a large language model with hidden state, according to some embodiments of this application. Figure 7As shown, the large language model jailbreak attack detection device 700 based on hidden state includes an extraction module 710, a projection module 720, and a detection module 730. In some embodiments of this application, the large language model jailbreak attack detection based on hidden state is performed by... Figure 2 The security analysis server shown performs the operation, wherein the apparatus includes:
[0137] Extraction module 710 is used to extract the hidden state of the user input to be detected in the set of security critical layers, wherein the hidden state is the output of the intermediate layer of the large language model, and the set of security critical layers is predetermined by the following method: constructing a jailbreak attack test set, applying random perturbation to the hidden state of each network layer of the large language model, evaluating the change in the success rate of jailbreak attacks before and after the perturbation, quantifying the importance of each network layer to security protection, and determining the set of security critical layers based on the importance;
[0138] The projection module 720 is used to project the hidden state of each layer onto the security vector corresponding to the layer, wherein the security vector is pre-extracted in the following way: for the set of security critical layers, the hidden state is extracted from the harmful instruction set and the harmless instruction set respectively, and the security vector of each security critical layer is extracted by principal component analysis.
[0139] The detection module 730 is used to stitch together all projection values to construct a feature vector, and input the feature vector into a pre-trained classifier to determine whether the user input to be detected is a jailbreak attack based on the output of the classifier.
[0140] In summary, the large language model jailbreak attack detection method and apparatus based on hidden layer states provided in the embodiments of this application quantify layer importance through perturbation experiments to locate security-critical layers. Because this technology can accurately identify the network layers most sensitive to security protection, it improves the accuracy and targeting of detection, overcoming the deficiency of insufficient detection accuracy at the input end. Security vectors are extracted through PCA analysis; because dimensionality reduction processing can efficiently capture the essential differences between harmful and harmless semantics, it significantly reduces computational overhead and solves the high overhead problem of model behavior analysis. Jailbreak attacks are detected through real-time projection and classifier techniques; because hidden layer states are analyzed in real-time during model inference, early interception is achieved, eliminating the lag in output-end detection. Finally, the synergistic integration of the above technologies forms a high-precision, low-overhead, and real-time jailbreak attack detection scheme, comprehensively improving the security of large language models.
[0141] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding descriptions in the foregoing device embodiments, and will not be repeated here.
[0142] Although the subject matter described herein is provided in the general context of execution on a computer system in conjunction with an operating system and applications, those skilled in the art will recognize that other implementations can also be executed in conjunction with other types of program modules. Generally, program modules include routines, programs, components, data structures, and other types of structures that perform specific tasks or implement specific abstract data types. Those skilled in the art will understand that the subject matter described herein can be practiced using other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframes, etc., and can also be used in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may reside on both local and remote memory storage devices.
[0143] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0144] It should be understood that the specific embodiments described above are merely illustrative or explanatory of the principles of this application and do not constitute a limitation thereof. Therefore, any modifications, equivalent substitutions, improvements, etc., made without departing from the spirit and scope of this application should be included within the protection scope of this application. Furthermore, the appended claims are intended to cover all variations and modifications falling within the scope and boundaries of the appended claims, or equivalent forms of such scope and boundaries.
Claims
1. A method for detecting jailbreak attacks using a large language model based on hidden state, characterized in that, include: Extract the hidden state of the user input to be detected in the set of security critical layers, wherein the hidden state is the output of the intermediate layer of the large language model. The set of security critical layers is predetermined by constructing a jailbreak attack test set, applying random perturbation to the hidden state of each network layer of the large language model, evaluating the change in the success rate of jailbreak attacks before and after the perturbation, quantifying the importance of each network layer to security protection, and determining the set of security critical layers based on the importance. The hidden state of each layer is projected onto the security vector corresponding to the layer, wherein the security vector is pre-extracted in the following way: for the set of security critical layers, the hidden state is extracted from the harmful instruction set and the harmless instruction set respectively, and the security vector of each security critical layer is extracted by principal component analysis. All projected values are concatenated to construct a feature vector, which is then input into a pre-trained classifier. The classifier output is used to determine whether the user input to be detected constitutes a jailbreak attack.
2. The method according to claim 1, characterized in that: The random perturbation is the addition of a noise vector that conforms to a Gaussian distribution, wherein the noise vector is sampled from a normal distribution with zero mean and adjustable variance.
3. The method according to claim 2, characterized in that: The random perturbation is implemented by the formula wherein: The hidden state generated by the large language model at the k-th network layer for the input instruction x. This refers to the hidden layer state after applying a perturbation to the network layer. The noise vector sampled from the Gaussian distribution. This is a scaling factor used to adjust the noise intensity.
4. The method of claim 3, wherein, The assessment of the change in the success rate of jailbreak attacks before and after the perturbation, and the quantification of the importance of each network layer to security protection, include: The importance score of the network layer is calculated using the formula wherein, For importance scores, The jailbreak success rate before performing perturbation operations on the large language model. The jailbreak success rate after performing a perturbation operation on the large language model is calculated. The scaling factor is mentioned.
5. The method of claim 4, wherein, Determining the set of safety critical layers based on the importance includes: Network layers whose importance scores exceed a preset threshold are selected to form the set of security-critical layers; or A predetermined number of network layers with the highest importance scores are selected to form the set of security-critical layers.
6. The method of claim 1, wherein, The extraction of the safety vector for each safety-critical layer through principal component analysis includes: Calculate the average hidden state activated by harmful and harmless instruction sets in the security-critical layer; The average hidden state is used to decentralize all hidden states extracted from the harmful and harmless instruction sets; The covariance matrix is calculated based on the decentralized hidden layer state; The covariance matrix is decomposed into eigenvalues, and the eigenvector corresponding to the largest eigenvalue is selected as the safety vector.
7. The method according to claim 6, characterized in that: The projection value is obtained through calculate, in, This refers to the hidden layer state. This refers to the average hidden layer state. This is the security vector.
8. The method of claim 1, wherein, The step of determining whether the user input to be detected is a jailbreak attack based on the classifier output includes: If the predicted value output by the classifier exceeds a preset threshold, the user input to be detected is determined to be a jailbreak attack.
9. A large language model escape attack detection device based on hidden layer state, characterized by, include: An extraction module is used to extract the hidden state of the user input to be detected in the set of security critical layers, wherein the hidden state is the output of the intermediate layer of the large language model, and the set of security critical layers is predetermined by the following method: constructing a jailbreak attack test set, applying random perturbation to the hidden state of each network layer of the large language model, evaluating the change in the success rate of jailbreak attacks before and after the perturbation, quantifying the importance of each network layer to security protection, and determining the set of security critical layers based on the importance; The projection module is used to project the hidden state of each layer onto the security vector corresponding to the layer. The security vector is pre-extracted in the following way: for the set of security critical layers, the hidden state is extracted from the harmful instruction set and the harmless instruction set respectively, and the security vector of each security critical layer is extracted by principal component analysis. The detection module is used to concatenate all projection values to construct a feature vector, and input the feature vector into a pre-trained classifier. Based on the classifier output, it determines whether the user input to be detected is a jailbreak attack.