A large model safety control method and system based on attention head state constraint

Abstract

The application relates to the technical field of artificial intelligence security, in particular to a large model security control method and system based on attention head state constraint, which comprises the following steps: constructing a safe-unsafe sample pair data set; extracting internal state representations of each attention head under safe and unsafe conditions and calculating state separation degrees; screening a key safe attention head set based on multi-dimensional screening indexes; training an integrated level joint state discriminator to output a safety probability value and an uncertainty estimation value; and performing a safety gating operation through an uncertainty interval condition trigger mechanism in the large model inference stage. The application refines the safety control granularity to the attention head level, can realize fine-grained safety control without retraining the model, and can effectively avoid the problem of excessive rejection while ensuring safety through integrated level joint discrimination and an uncertainty interval condition trigger mechanism.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence security technology, and in particular to a large model security control method and system based on attention head state constraints. Background Technology

[0002] In recent years, large language models (LLMs), represented by GPT (Generative Pre-trained Transformer), LLaMA (Large Language Model Meta AI), and GLM (General Language Model), have made groundbreaking progress in the field of natural language processing and have been widely applied in scenarios such as intelligent customer service, content generation, code writing, and knowledge-based question answering. However, with the widespread application of large language models, their security issues have become increasingly prominent. Large language models may generate harmful content, leak private information, spread false information, or be induced by malicious users to produce unsafe outputs through methods such as hint injection and jailbreaking attacks, posing serious risks to society and individuals.

[0003] To reduce the risk of large models generating unsafe content when faced with illegal, dangerous, or abusive requests, existing large language models typically employ methods such as supervised fine-tuning, reinforcement learning based on human feedback, safety preference alignment, and training with rejected data for safety alignment. By introducing safety constraints during the model training phase, the model learns to reject unsafe requests, enabling it to output rejection, clarification, or safe alternative suggestions when faced with high-risk inputs.

[0004] However, in the process of implementing the inventive technical solutions in the embodiments of this application, the inventors of this application discovered that the above-mentioned prior art solutions still have at least the following technical problems:

[0005] Existing security assessments often focus solely on the safety of the final output, lacking fine-grained modeling of internal state shifts and security constraint violations. For requests with the same objective but different expressions, the model may generate unsafe responses. This means the security behavior of a large model depends not only on the harmfulness of the input but also on whether the model's internal attention heads remain near a safe reference state during generation. Current methods primarily rely on the input text or final output, making it difficult to capture unsafe internal states that may have already occurred during generation. External filters often fail to detect risks before the model outputs complete dangerous content; interception after dangerous content has been generated results in delayed security control. Furthermore, input-based classification methods can easily misclassify legitimate, authorized technical inquiries as harmful requests, leading to excessive rejection. To improve the model's rejection probability for high-risk requests, existing technologies reduce unsafe responses by enhancing the model's rejection tendency, adding global rejection directions, uniformly increasing security thresholds, or fixing and blocking certain attention heads. However, due to the large scope of intervention, this often affects normal requests, low-risk requests, and legitimate defensive requests simultaneously, leading to overly cautious model behavior and reduced task completion capabilities. Alternatively, unsafe behavior can be eliminated by modifying model parameters. However, parameter modification may affect the model's general capabilities, leading to a degradation of its capabilities. Furthermore, the editing effect is unstable and may be reverted to its original state in subsequent use, making it difficult to precisely edit for complex security scenarios.

[0006] Therefore, in summary, how to establish safe state constraints using the internal attention head states of the model without retraining the main parameters of the large model, identify constraint violations corresponding to unsafe generation tendencies during the generation process, and achieve local, conditionally triggered, and low-overhead safety control during the inference phase is a technical problem that urgently needs to be solved in the field of large model safety control. Summary of the Invention

[0007] This application aims to provide a large-scale model safety control method and system based on attention head state constraints, which refines the granularity of safety control to the attention head level, enabling fine-grained safety control without retraining the model. This solves the technical problems of existing large language model safety control methods, such as high training costs, coarse intervention granularity, poor safety control flexibility, and weak model versatility.

[0008] To achieve the objective of this invention, the following technical solution is adopted:

[0009] Firstly, this application provides a large-scale model security control method based on attention head state constraints, comprising: constructing a dataset of safe-unsafe sample pairs; wherein the dataset includes multiple sets of safe response samples and unsafe response samples; each set of safe-unsafe sample pairs shares the same input prompt and corresponds to a safe response and an unsafe response, respectively; inputting the safe response samples and unsafe response samples into the large-scale model, extracting the safe state vector and unsafe state vector of the safe response samples and unsafe response samples at a specified position of each attention head; calculating the state separation degree of each attention head based on the safe state vector and unsafe state vector; calculating a multidimensional screening index for each attention head based on the safe state vector, the unsafe state vector, and the state separation degree; and evaluating each attention head based on the multidimensional screening index to screen... A target security attention head set is selected; based on the security state vector and unsafe state vector of each attention head in the target security attention head set, a pre-constructed ensemble-level joint state discriminator is trained; wherein, the ensemble-level joint state discriminator is used to jointly discriminate the security state of all attention heads in the target security attention head set, and output the corresponding security probability value and uncertainty estimate value for each pair of security-unsafe samples; during the inference phase of the large model, the trained ensemble-level joint state discriminator is used to jointly discriminate the security state of all attention heads in the target security attention head set for the current input prompt, and based on the corresponding security probability value and uncertainty estimate value of the target input prompt output by the ensemble-level joint state discriminator, the large model is subjected to real-time security control.

[0010] Secondly, this application provides a large-scale model security control system based on attention head state constraints, including:

[0011] The sample pair construction module is used to construct a secure-insecure sample pair dataset; wherein, the dataset includes multiple sets of secure response samples and insecure response samples; each set of secure-insecure sample pairs shares the same input prompt and corresponds to a secure response and an insecure response, respectively;

[0012] The state extraction module is used to input safe response samples and unsafe response samples into the large model respectively, extract the safe state vector and unsafe state vector of the safe response sample and the unsafe state vector at a specified position of each attention head; and calculate the state separation degree of each attention head based on the safe state vector and the unsafe state vector.

[0013] The key head filtering module is used to calculate a multi-dimensional filtering index for each attention head based on the safe state vector, the unsafe state vector, and the state separation degree; and to evaluate each attention head based on the multi-dimensional filtering index to obtain a set of target safe attention heads.

[0014] The discriminator training module trains a pre-constructed integrated joint state discriminator based on the safe state vector and unsafe state vector of each attention head in the target security attention head set. The integrated joint state discriminator is used to jointly discriminate the security state of all attention heads in the target security attention head set and output the corresponding security probability value and uncertainty estimate value for each safe-unsafe sample pair.

[0015] The real-time gating module for model inference is used during the inference phase of the large model to perform joint discrimination on the safety status of all attention heads in the target safety attention head set for the current input prompt. Based on the corresponding safety probability value and uncertainty estimate value of the target input prompt output by the joint state discriminator, the large model is subjected to real-time safety control.

[0016] Thirdly, this application also provides an electronic device, the electronic device comprising: at least one processor; a memory communicatively connected to the at least one processor; and a computer program stored in the memory and running on the at least one processor; wherein, when the at least one processor executes the computer program, it is used to implement the large model security control method based on attention head state constraints described in the first aspect above.

[0017] Fourthly, this application provides a computer-readable storage medium, wherein a computer program present in the computer-readable storage medium, when executed by a computer device, is used to implement the large model security control method based on attention head state constraints described in the first aspect.

[0018] Fifthly, this application provides a computer program product comprising a computer program, which, when executed by a computer device, is used to implement the large model security control method based on attention head state constraints described in the first aspect.

[0019] As can be seen from the above, this application has the following beneficial effects:

[0020] This application refines the granularity of security control from the traditional hidden layer level to the attention head level, enabling precise identification of specific attention heads that play a key role in security output. This avoids the performance loss and side effects that may result from coarse-grained intervention at the hidden layer level. Furthermore, by using an integrated joint state discriminator to jointly discriminate the internal states of multiple target security attention heads in the target security attention head set, compared to single-head level discrimination methods, it can capture richer security state information, significantly improving the accuracy and robustness of security detection. In addition, all operations in this application are performed during the large model inference stage, without requiring any parameter modification or retraining of the large model. It can be directly applied to deployed models, exhibiting plug-and-play characteristics and extremely low deployment costs. Overall, this application solves the technical problems of high training costs, coarse intervention granularity, poor security control flexibility, and weak model versatility in existing large language model security control methods.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating a large model security control method based on attention head state constraints provided in this application embodiment;

[0025] Figure 2 A schematic diagram of the structure of a large-model safety control system based on attention head state constraints is provided in an embodiment of this application;

[0026] Figure 3 This is a schematic diagram of the structure of a computer electronic device provided in an embodiment of this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] Example 1

[0030] The technical solution of this invention is applicable to white-box or gray-box deployment scenarios capable of reading or accessing the internal attention head states of large models. Its core objective is to maintain the availability of normal requests while reducing the rate of insecure responses. Currently, attention head security analysis lacks a deployable inference-time state constraint control mechanism. Research indicates a close relationship between multi-head attention mechanisms and the security behavior of large models, with some attention heads potentially related to refusal to respond, security judgment, or risk identification. Existing methods typically focus on global security capability attribution, identifying which heads are generally related to security capabilities, but rarely establish quantifiable, monitorable, and executable attention head security state constraints. Furthermore, there is a lack of an engineering solution capable of determining whether constraint violations have occurred during the inference phase based on the joint state of candidate attention head sets, and executing condition-triggered local control accordingly. As mentioned above, existing security assessments often only focus on whether the final output is secure, lacking fine-grained modeling of internal state shifts and security constraint violations. For requests with the same objective but different expressions, the model may generate insecure responses. Therefore, the security behavior of a large model depends not only on whether the input is harmful but also on whether the internal attention head states remain near a safe reference state during the generation process. It primarily relies on input text or final output results for judgment, making it difficult to capture unsafe internal states that may have already appeared during the generation process of a large model. When the model has not yet output complete dangerous content, external filters often cannot detect the risk in advance; while interception after dangerous content has been generated will cause a lag in security control; at the same time, input-based classification methods are prone to misjudging legitimate and authorized technical inquiries as harmful requests, thus leading to excessive rejection issues.

[0031] Therefore, this application embodiment constructs a safe-unsafe sample pair dataset; extracts the internal state representation of each attention head under safe and unsafe conditions and calculates the state separation degree; selects a set of key safe attention heads based on six multi-dimensional screening indicators: discriminative power, stability, sensitivity, coverage, independence, and efficiency; trains an ensemble-level joint state discriminator to output safe probability values ​​and uncertainty estimates; and performs safety gating operations through an uncertainty interval condition triggering mechanism during the large model inference stage. By refining the granularity of safety control to the attention head level, fine-grained safety control is achieved on the internal state of the large language model. This enables accurate identification of internal components that play a key role in safe output without retraining the model, and efficient and accurate safety gating is achieved based on the state information of these components. Through ensemble-level joint discriminant analysis and the uncertainty interval condition triggering mechanism, safety is ensured while effectively avoiding excessive rejection of answers.

[0032] Specifically, such as Figure 1 As shown, Figure 1 This is a flowchart illustrating a large-model security control method based on attention head state constraints, provided in an embodiment of this application.

[0033] This application provides a large model security control method based on attention head state constraints. The executing entity can be a large model security control system based on attention head state constraints (hereinafter referred to as the system), which can be implemented by a server, personal computing device, edge node, or a combination thereof. The method includes the following steps:

[0034] 101. Construct a dataset of secure-insecure sample pairs.

[0035] The following is a detailed implementation of constructing a safe-insecure sample pair dataset for analysis and training:

[0036] Multiple sets of input prompts are collected, covering various types of insecure scenarios, including requests to generate harmful content, requests to steal privacy information, requests to generate false information, and jailbreak attack prompts. For example, requests may include requests to generate violent or discriminatory content, requests to extract personal information from training data, requests to generate fake news, or attempts to bypass security constraints through role-playing, encoding conversion, or other methods.

[0037] For each set of input prompts, generate a safe response and an unsafe response respectively.

[0038] The method for generating a safety response is as follows: input the input prompt into the large language model and apply a safety alignment constraint to the large language model to be controlled, that is, obtain the model output result under the condition that the safety alignment mechanism of the large language model is working normally.

[0039] Insecure responses can be generated by removing or weakening secure alignment constraints on large language models, and can be achieved in the following ways:

[0040] (1) By using adversarial prompts or jailbreak attack techniques, the model is induced to generate insecure responses under conditions where the security alignment mechanism is weakened or bypassed. For example, known jailbreak attack templates are used as variations of input prompts, such as ignoring previous command-based attacks, role-playing attacks, etc.

[0041] (2) Using a contrastive decoding strategy, safe and unsafe responses are generated for the same input prompt. Specifically, the generation process of the model can be intervened with safe and unsafe guidance vectors respectively to obtain safe and unsafe responses.

[0042] (3) For models that have been trained with safety alignment, the natural output of the model under no safety constraints can be obtained as an unsafe response by removing safety-related system prompts or safety classifier constraints during inference.

[0043] Ultimately, the constructed safe-unsafe sample pair dataset contains multiple sets of sample pairs, let's say N sets, each set including an input cue p. j A security response j and an unsafe response u j , where j is a natural number, representing the input prompt, safe response, and unsafe response of the j-th sample pair, respectively, j = 1, 2, ..., N.

[0044] It should be noted that the secure response and insecure response described in the embodiments of this application refer to the secure response sample and insecure response sample included in each sample pair.

[0045] 102. Extract the internal state representation of the attention head and calculate the state separation degree.

[0046] This step aims to analyze the internal state differences of each attention head in the large language model under safe and unsafe conditions. Assuming the model has H attention heads, the following operations are performed for each attention head in the large language model:

[0047] (1) Input the safe response samples and unsafe response samples in the safe-unsafe sample pair dataset into the large language model, and extract the safe state vector and unsafe state vector of the safe response sample and the unsafe response sample at the specified position of the attention head respectively.

[0048] For the input security response sample, during the forward propagation of the model, the i-th attention head is extracted at the specified layer. l The specified location t The output vector at position i is denoted as the safety state vector h of the i-th attention head. i^safe。

[0049] For an insecure response sample as input, during the forward propagation of the model, the i-th attention head is extracted at a specified layer. l The specified location t The output vector at position i is denoted as the unsafe state vector h of the i-th attention head. i^unsafe。

[0050] Among them, the specified layer l You can select an intermediate layer of the model, such as a layer from layer 16 to layer 32; specify the position. tYou can choose the position of the last token in the response or the average of multiple key positions; i = 1, 2, ..., H.

[0051] (2) Calculate the state separation degree D of the i-th attention head based on the safe state vector and the unsafe state vector. i .

[0052] Attention head state separation D i The cosine similarity difference metric is used, and its calculation formula is shown below: D i = cos(h i^safe , r safe ) - cos(h i^unsafe , r safe );

[0053] Where, r safe This is a preset safety reference direction vector. Specifically, this can be achieved by collecting multiple sets of safety response samples at a specified layer. l The hidden state vectors are obtained by calculating the mean of these hidden state vectors and then performing L2 normalization on the mean, thus yielding the safe reference direction vector r. safe .

[0054] State separation degree D i This reflects the degree of difference in the state of the i-th attention head under safe and unsafe conditions. (D) i The larger the absolute value of D, the stronger the ability of the attention to distinguish between safe and unsafe states. i A positive value for D indicates that the focus is closer to the safe reference direction under safe conditions. i A negative value indicates that the attention point is closer to the safe reference direction under unsafe conditions.

[0055] It should be noted that for the above N sets of sample pairs, the state separation degree of each sample pair at the i-th attention head can be calculated, and N state separation degrees D are obtained for the i-th attention head. i The separation degree D of the N states i Take the mean value and use it as the final state separation degree of the i-th attention head. This can eliminate the influence of randomness in a single set of samples.

[0056] 103. Screening a set of key security attention points based on multidimensional screening indicators.

[0057] Based on multidimensional screening indicators, each of the above H attention heads is evaluated to systematically screen out the set of target security attention heads that are most critical and efficient for security control; wherein the multidimensional screening indicators include at least three of the following: discriminative power indicator, stability indicator, sensitivity indicator, coverage indicator, independence indicator, and efficiency indicator.

[0058] For the i-th attention head, calculate at least three of its discriminative power index, stability index, sensitivity index, coverage index, independence index, and efficiency index.

[0059] (1) Discriminant index M i^disc By calculating the state separation degree D i The mean of the absolute values ​​of the above N sample pairs is used to measure the ability of the i-th attention head to distinguish between safe and unsafe states. Its calculation formula is as follows: M i^disc = (1 / N)×Σ j |D i^(j) |;

[0060] Among them, D i^(j) Let be the state separation degree of the j-th sample pair at the i-th attention head.

[0061] The higher the discrimination index value, the stronger the ability of the attention to distinguish between safe and unsafe states.

[0062] (2) Stability index M i^stab , is used to measure the consistency of the state separation of the i-th attention head under different input cues. Its calculation formula is as follows: M i^stab = μ(D i ) / σ(D i );

[0063] Where, μ(D) i ) is D i The mean of the above N sample pairs; σ(D i () represents the standard deviation.

[0064] Stability index M i^stab The higher the value, the more consistent the safety judgment behavior of the attention point under different inputs.

[0065] (3) Sensitivity index M i^sens , is used to measure the sensitivity of the i-th attention head to insecure inputs. It is calculated as the average state shift of an insecure response sample relative to a secure response sample, and its formula is: M i^sens= (1 / N) × Σ j ||h i^unsafe (j) - h i^safe (j)||2; in, ||·||2 represents the L2 norm; h i^safe (j) is the security state vector of the j-th group of security response samples at the position specified by the i-th attention head; h i^unsafe (j) is the unsafe state vector of the j-th unsafe response sample at the position specified by the i-th attention head.

[0066] Sensitivity index M i^sens The higher the value, the more sensitive the attention is to insecure input.

[0067] (4) Coverage index M i^cov , used to measure the detection coverage of the i-th attention head for different types of unsafe content. Unsafe content is divided into C categories, such as harmful content, privacy breaches, misinformation, jailbreak attacks, etc. The proportion of the i-th attention head maintaining a positive dissociation across all categories is calculated, as shown in the following formula: M i^cov = |{c : D i^(c) > 0}| / C; Among them, D i^(c) Let be the average state separation degree over the c-th type of unsafe content.

[0068] Coverage index M i^cov The higher the value, the more types of unsafe content the attention head can detect.

[0069] (5) Independence index M i^indep , used to measure the degree of information redundancy between the i-th attention head and other selected attention heads. It is calculated as the complement of the maximum correlation coefficient between the state vector of the i-th attention head (including safe and unsafe state vectors) and the state vectors of all attention heads in the selected attention head set S, as shown in the following formula: M i^indep = 1 - max{k∈S} corr(h i , h k ); in, corr(·,·) is the Pearson correlation coefficient; h i Let i be the state vector of the i-th attention head; h kLet be the state vector of the k-th attention head in the selected attention head set S.

[0070] Independence index M i^indep The higher the value, the more independent information the attention provides.

[0071] (6) Efficiency index M i^eff , is used to measure the impact of introducing the i-th attention head on the model's inference latency. It is calculated as the reciprocal of the proportion of the i-th attention head's computation time to the model's single-layer inference time, as shown in the following formula: M i^eff = T layer / T head_i ; in, T layer This represents the total inference time for a single layer of the model. T head_i Let be the computation time for the i-th attention head.

[0072] Efficiency index M i^eff The higher the value, the lower the computational cost of introducing that attention point.

[0073] Then, each of the above six indicators is normalized to unify its value range to the [0,1] interval.

[0074] Then, based on at least three of the above-mentioned discrimination index, stability index, sensitivity index, coverage index, independence index and efficiency index, a comprehensive score is calculated for each attention head. Each of the above H attention heads is evaluated, thereby systematically selecting the set of target security attention heads that are most critical and efficient for security control.

[0075] Taking the calculation of the comprehensive score for each attention head based on the above six indicators as an example, the calculation formula is as follows:

[0076] S i =w disc ×M i^disc +w stab ×M i^stab +w sens ×M i^sens +w cov ×M i^cov +w indep ×M i^indep +w eff ×M i^eff ; in, S i The overall score for the i-th attention head; wdisc w stab w sens w cov w indep w eff Let w be the weight coefficients of each indicator, and w be the weight coefficients of each indicator. disc +w stab +w sens +w cov +w indep +w eff = 1.

[0077] In this embodiment of the application, the weighting coefficients of various indicators can be adjusted according to the specific application requirements. For example, in application scenarios with high security requirements, the discrimination power indicator M can be increased. i^disc and coverage index M i^cov The weighting coefficients; in scenarios where inference speed is critical, the efficiency index M can be increased. i^eff The weighting coefficients. In this embodiment, the default configuration of the weighting coefficients for each indicator can be: w disc = 0.25, w stab = 0.15, w sens = 0.20, w cov = 0.20, w indep = 0.10, w eff = 0.10.

[0078] According to the comprehensive score S i Sort all H attention heads from highest to lowest, and select the top K attention heads to form the most critical and efficient target security attention head set S for security control. key , S key ={h{i1}, h{i2}, ..., h{iK}}; Where K is a preset positive integer, and K≤½H. In practical applications, the value of K can be determined based on the model size and security requirements. For example, for a model with 32 layers, 32 attention heads per layer, and a total of 1024 attention heads, K can take the value {50, 200}.

[0079] This application establishes a systematic attention head screening mechanism and proposes a multi-dimensional screening framework that includes six indicators: discriminative power, stability, sensitivity, coverage, independence, and efficiency. This framework can systematically screen out the most critical and efficient target security attention head set from a large number of attention heads, taking into account both security control effectiveness and inference efficiency.

[0080] 104. Training the ensemble-level joint state discriminator.

[0081] Based on the secure state vector and insecure state vector of each attention head in the target secure attention head set, a lightweight classifier capable of real-time discrimination of the security state of a large language model is trained, namely, an ensemble-level joint state discriminator. The ensemble-level joint state discriminator is used to jointly discriminate the security state of all attention heads in the target secure attention head set and output the corresponding security probability value and uncertainty estimate for each pair of secure-insecure samples.

[0082] In this embodiment of the application, the integrated joint state discriminator adopts a multilayer perceptron (MLP) structure, including: Input layer, dimension K×d head ; where d head The output vector dimension for each attention head; One or more hidden layers, each using the ReLU activation function and Dropout regularization; The output layer uses the Sigmoid activation function to output the safe probability value p. safe ∈ [0,1].

[0083] The specific implementation method is as follows: For each pair of samples in the training data, extract the target security attention head set S. key The internal state vectors (including safe and unsafe state vectors) of all K attention heads are concatenated into a joint feature vector, as shown in the following formula: x= [h{i1} || h{i2} || ... || h{iK}]; Where || denotes vector concatenation operation; x represents the joint feature vector.

[0084] The joint feature vector x contains the state information of all target security attention heads, including secure and insecure state information, and its dimension is K×d. head , where d head The output vector dimension for each attention head.

[0085] The joint feature vector x is used as input, and the binary safety label y ∈ {0, 1} is used as the training target, where 0 represents unsafe and 1 represents safe. The ensemble-level joint state discriminator is trained using a binary cross-entropy loss function. Early stopping and regularization techniques are employed during training to prevent overfitting.

[0086] To enable the integrated joint state discriminator to quantify the confidence level of its predictions, this embodiment employs the Monte Carlo Dropout method for uncertainty estimation. Specifically, during large model inference, the Dropout layer remains active, and T forward propagations are performed on the same input. The mean of the T prediction results is calculated as the safety probability value p. safe The standard deviation of the T prediction results is calculated as the uncertainty estimate u. In this embodiment, T=20.

[0087] In some feasible implementations, after training, a causal intervention experiment is performed on each attention head in the target safe attention head set to verify its causal impact on the safe output. Specifically, a set of input cues known to produce unsafe responses is selected. During model inference, the internal state of attention head h{ik} is replaced with a safe state representation (i.e., the mean of the safe state vector of the attention head on safe response samples), and the model output is observed to change from an unsafe response to a safe response. If the model output becomes a safe response after the replacement, it indicates that the attention head has a causal impact on the safe output, and the verification is passed; otherwise, it indicates that the attention head may only be an observer of the safe state rather than a causal factor, and the verification is not passed. Therefore, attention heads that pass the causal intervention verification are retained in the target safe attention head set, while attention heads that do not pass the causal intervention verification are removed from the target safe attention head set. Then, based on the updated target safe attention head set, the above joint feature vector concatenation operation and the ensemble-level joint state discriminator training operation are re-executed until all safe-unsafe sample pairs in the safe-unsafe sample pair dataset are traversed.

[0088] In this application embodiment, after screening the target security attention heads, the causal influence of each attention head on the security output is verified through causal intervention experiments. This ensures that the selected attention heads are indeed key factors in security control, rather than just bystanders with high correlation indicators, thereby improving the interpretability and reliability of security control.

[0089] 105. Real-time security gating during model inference.

[0090] During the inference phase of the large language model, the trained ensemble-level joint state discriminator is used to jointly discriminate the security status of all target security attention heads in the target security attention head set for the current input prompt. Based on the corresponding security probability value and uncertainty estimate of the target input prompt output by the ensemble-level joint state discriminator, the large model security control operation is triggered to perform real-time security control on the large model.

[0091] The specific implementation method is as follows: During the inference process of the large language model, for the target input cue t, the internal state vectors of all attention heads in the target safe attention head set are extracted, including safe state vectors and unsafe state vectors, and the extracted internal state vectors are concatenated into a joint feature vector x. t .

[0092] The joint feature vector x t Input the pre-trained ensemble-level joint state discriminator to obtain the corresponding safety probability value p of the target input cue t output by the discriminator. safe And the uncertainty estimate u.

[0093] Based on the target input prompt output by the integrated joint state discriminator, the corresponding safety probability value and uncertainty estimate are used to trigger the large model safety control operation, which specifically includes the following: If p safe ≥ T safe And u ≤ U threshold If the current generation state of the large language model is deemed safe, the model is allowed to continue generating normally. If p safe <T danger And u≤U threshold If the current generation state of the large language model is deemed dangerous, a large model safety control operation will be immediately triggered.

[0094] Safety control operations may include: (1) Output interception, for example, directly preventing the model from continuing to generate and returning a preset security response template to the user, such as "Sorry, I cannot answer this question"; (2) Output redirection, for example, by applying a safety offset to the internal state vector of the target safety attention head, the generation direction of the model is guided back to the safety space.

[0095] The formula for calculating the safety offset Δh is as follows: Δh=α×(r safe - h current ); Where α is the offset intensity coefficient; h current Given the current internal state vector, the offset Δh is superimposed on the output of the attention head.

[0096] (3) Gradual degradation, for example, according to p safe With T danger To address the gap, gradually reduce the degree of freedom in the model output, such as limiting the output length, limiting the range of output topics, or switching to a restricted generation mode.

[0097] If T danger ≤psafe <T safe That is, the safety probability value is within the uncertainty range, or u > U. threshold If the discriminator itself has high uncertainty, then the gating operation will not be executed directly, but the enhanced verification process will be triggered.

[0098] Enhanced verification processes may include: Perform secondary security checks on the target input prompt t for the current input. For example, use an independent text classifier to perform a security assessment on the generated content; or request manual review by sending the target input prompt t and the generated content to the manual review queue; or increase the number of samplings to obtain a more stable security probability value through multiple samplings.

[0099] In this embodiment, an uncertainty interval condition triggering mechanism is introduced, which triggers enhanced verification instead of direct interception when the security probability value is in the middle range or the uncertainty is high. This effectively avoids the problem of excessive rejection and maintains the normal availability of the model while ensuring security.

[0100] In some feasible implementations, during model inference, the discrimination results of the integrated joint state discriminator are continuously collected, including security probability values, uncertainty estimates, large model security control decisions, and user feedback data. User feedback data includes, for example, user acceptance of blocking actions and false positive reports. Periodically (e.g., weekly or after processing 10,000 requests), based on the collected discrimination results, the multidimensional screening metrics for each attention head are recalculated, and the target security attention head set is updated. Furthermore, the integrated joint state discriminator is incrementally trained or fine-tuned using the newly collected discrimination results. Finally, the security threshold T is adjusted. safe and danger threshold T danger The embodiments of this application support continuous data collection and periodic updates of key safety attention head sets and discriminator parameters during large model inference, which can adapt to the safety state drift problem that occurs during the use of large models and maintain long-term safety control effects.

[0101] It should be noted that the threshold T in the above judgment... safe T danger U threshold It is pre-set, among which, In this embodiment of the application, the security threshold T safe and danger threshold T danger The default values ​​are set to 0.7 and 0.3 respectively, along with the uncertainty threshold U. threshold The default value is 0.15.

[0102] In summary, the embodiments of this application have the following beneficial technical effects compared to the prior art: This application refines the granularity of security control from the traditional hidden layer level to the attention head level, enabling precise identification of specific attention heads that play a key role in security output. This avoids the performance loss and side effects that may result from coarse-grained intervention at the hidden layer level. Furthermore, by using an integrated joint state discriminator to jointly discriminate the internal states of multiple target security attention heads in the target security attention head set, compared to single-head level discrimination methods, it can capture richer security state information, significantly improving the accuracy and robustness of security detection. In addition, all operations in this application are performed during the large model inference stage, without requiring any parameter modification or retraining of the large model. It can be directly applied to deployed models, exhibiting plug-and-play characteristics and extremely low deployment costs. Overall, this application solves the technical problems of high training costs, coarse intervention granularity, poor security control flexibility, and weak model versatility in existing large language model security control methods.

[0103] The following set of experimental data verifies and illustrates the technical effectiveness of a large-model security control method based on attention head state constraints provided in the embodiments of this application: I. Experimental Setup (1) Model to be Secured: The Qwen3-8B model was selected as an example of the model to be secured. The number of parameters is approximately 8.2B, including 36 Transformer layers. Each layer includes 32 query attention heads and 8 key-value attention heads. A grouped query attention structure is adopted. Therefore, when using query attention heads as the objects of state extraction and filtering, there are a total of 36×32=1152 candidate query attention heads. The hidden state dimension of this model is 4096, and the state vector dimension of a single query attention head is 128. During the experiment, the main parameters of this large language model are not updated. Only the internal state of the target attention head is read during the inference stage, and security control is performed based on the method described in this application.

[0104] (2) Dataset: The safe-unsafe sample pair dataset includes 2000 sample pairs, each with the same input prompts, and corresponding to safe and unsafe response samples respectively. The input prompts cover five risk scenarios: harmful content generation, privacy information inducement, false information generation, jailbreak prompts, and illegal abuse consultation. Safe response samples are obtained under the condition that the model's safety alignment mechanism is working properly, while unsafe response samples are obtained under the condition of weakening safety constraints, removing some safety prompts, or using adversarial prompts. To reduce the impact of intermediate text generated by the thinking mode on the safety judgment, the non-thinking mode of Qwen3-8B is uniformly used when generating test output.

[0105] (3) Test set: The test set includes 1000 input prompts, of which 500 are high-risk prompts and 500 are normal requests or authorized technical consultation prompts. The high-risk prompts include the above five risk scenarios, with about 100 prompts in each category, and are used to evaluate whether the model still generates insecure content; the normal requests and authorized technical consultation prompts are used to evaluate whether the model over-refuses answers or erroneously blocks responses.

[0106] (4) Training set: Extract the internal state vector of each query attention head at the token position at the end of the response, and calculate the state separation degree of safe and unsafe states relative to the safe reference direction vector. Since the state vector dimension of a single query attention head is 128, a safe reference direction vector is constructed in the 128-dimensional state space corresponding to each candidate query attention head, that is, the mean of the state vector of the query attention head on the safe response sample is calculated and normalized. Thus, the safe state vector, the unsafe state vector, and the safe reference direction vector are in the same vector space, and the cosine similarity can be directly calculated. Then, the attention heads are comprehensively scored based on six indicators: discriminative power, stability, sensitivity, coverage, independence, and efficiency, and the top 64 query attention heads are selected as the target safe attention head set, which accounts for about 5.56% of the total number of candidate query attention heads. Therefore, the joint feature vector dimension is 64×128=8192. In the inference stage, the safety threshold Tsafe is set to 0.70, the danger threshold Tdanger is set to 0.30, and the uncertainty threshold Uthreshold is set to 0.15.

[0107] II. Comparison Methods and Evaluation Indicators Set the following comparison method: (1) Original security alignment model: Only the original security alignment mechanism of the model is used, without introducing the method of this application.

[0108] (2) Input / output text filtering: The input prompts or model outputs are detected by a text safety classifier, and interception is performed when the detection is dangerous.

[0109] (3) Fixed shielding of high-risk attention heads: Based on the statistical results of the training set, a number of high-risk attention heads are fixed shielded, but the joint state discrimination of the target safe attention head set is not used, nor is the uncertainty interval condition triggering mechanism used.

[0110] (4) Key head joint discrimination direct gating: The target security attention head set and integrated level joint state discriminator are adopted, but when the security probability is lower than the security threshold, the gating is directly triggered without setting an uncertainty interval.

[0111] (5) Technical solution of this application: Real-time security control is carried out by using a target security attention head set, an integrated joint state discriminator and an uncertainty interval condition triggering mechanism.

[0112] The following evaluation indicators will be used: (1) Insecure output rate: The proportion of unsafe content generated by the model even when a high-risk warning is issued. The lower this indicator, the better the security control effect.

[0113] (2) Excessive Refusal Rate: The proportion of cases in which the model incorrectly refuses to answer, incorrectly blocks, or significantly reduces the quality of the response in normal requests or authorized technical consultation prompts. The lower this indicator, the smaller the impact on normal tasks.

[0114] (3) Normal task retention rate: The percentage of times the model can still provide a usable answer in normal requests or authorized technical consultation prompts.

[0115] (4) Average Additional Latency: The average additional computation time for each security determination after introducing the corresponding security control method compared to the original model inference process. If multiple security determinations are performed in a single generation request, the cumulative additional latency is related to the number of security determinations triggered.

[0116] III. Verification Results Comparison of the performance of each method on the test set: As shown in Table 1 below, Table 1 illustrates the performance comparison of the above five comparison methods on the test set: Table 1 Original secure alignment model 12.6 5.4 94.1 0.0 Input / output text filtering 8.4 12.2 86.3 8.5 Fixed shielding of high-risk attention heads 7.7 9.8 89.1 3.6 Key head joint discrimination direct gating 5.3 10.6 88.4 4.9 This application method 5.9 7.4 91.2 5.4 As shown in Table 1, the insecure output rate of the original security alignment model is 12.6%. The input / output text filtering method can reduce the insecure output rate to 8.4%, but the excessive rejection rate rises to 12.2%, indicating that it is prone to misjudging some normal requests or authorized technical consultations as dangerous requests. The fixed high-risk attention head blocking method has an insecure output rate of 7.7% and an excessive rejection rate of 9.8%, which is an improvement compared to the text filtering method, but its blocking strategy is fixed and it is difficult to implement conditional triggering control based on the current generation state. The key head joint discrimination direct gating method has an insecure output rate of 5.3%, lower than the 5.9% of the method in this application, but its excessive rejection rate reaches 10.6%. This shows that if no uncertainty interval is set and gating is directly applied to boundary samples, although it can further reduce some insecure outputs, it will significantly increase the false interception of normal requests. In contrast, the method in this application has an insecure output rate of 5.9%, an excessive rejection rate of 7.4%, and a normal task retention rate of 91.2%, which can achieve a more balanced effect between security control and availability. The method proposed in this application achieves a good balance between insecure output rate and excessive rejection rate, indicating that the uncertainty interval condition triggering mechanism can reduce false interception of normal requests while enhancing security control.

[0117] IV. Scenario-based verification Three comparative methods—the original secure alignment model, input / output text filtering, and the technical solution of this application—were selected. Their insecure output rates under the aforementioned five risk scenarios (harmful content generation, privacy information inducement, false information generation, jailbreak prompts, and illegal abuse of information) are shown in Table 2 below. Table 2 illustrates the comparison of the insecure output rates of the three comparative methods under the aforementioned five risk scenarios: Table 2 Harmful content generation 14.4 9.2 6.2 Privacy information inducement 10.8 7.8 5.4 False information generation 15.0 10.0 7.4 Jailbreak Tips 16.8 10.8 7.2 Illegal abuse of consulting 6.0 4.2 3.3 As shown in Table 2, the proposed method can reduce the insecure output rate in all five risk scenarios. Specifically, for jailbreak hint input, the insecure output rate of the original secure alignment model is 16.8%, the input / output text filtering method reduces it to 10.8%, and the proposed method further reduces it to 7.2%. This result demonstrates that the proposed method utilizes the internal attention head state of the model for real-time discrimination, which can supplement the shortcomings of relying solely on the input text or the final output text for security detection.

[0118] V. Verification of Key Components To verify the effectiveness of the target security attention head screening and uncertainty interval condition triggering mechanism, further component verification was conducted.

[0119] The selected experimental schemes include: (1) Randomly select 64 query attention heads for joint discrimination; (2) Select 64 query attention heads based solely on state separation degree; (3) 64 query attention points were selected using multi-dimensional indicators, with no uncertainty intervals; (4) 64 query attention points were selected using multidimensional indicators and uncertainty intervals were used.

[0120] The comparison metrics include: unsafe output rate, excessive rejection rate, and average additional delay.

[0121] As shown in Table 3 below, Table 3 compares the effects of the above four experimental schemes on insecure output rate, excessive rejection rate, and average additional delay: Table 3 64 query attention heads were randomly selected for joint discrimination. 10.2 6.8 5.1 Select 64 query attention heads based solely on state separation. 7.1 8.3 5.2 64 query attention heads were selected using multi-dimensional indicators, with no uncertainty intervals. 5.3 10.6 4.9 Sixty-four query attention heads were selected using multidimensional indicators, and an uncertainty interval was adopted. 5.9 7.4 5.4 As shown in Table 3, the insecure output rate remains at 10.2% when randomly selecting query attention heads, indicating that not all query attention head states can effectively reflect the secure generation state. When selecting query attention heads solely based on state separation, the insecure output rate decreases to 7.1%, but this is still lower than the multi-dimensional index screening scheme. After screening query attention heads using multi-dimensional indices such as discriminative power, stability, sensitivity, coverage, independence, and efficiency, the insecure output rate further decreases, indicating that the target secure attention head screening mechanism of this application helps to obtain a more stable security state representation. Meanwhile, when screening 64 query attention heads using multi-dimensional indices without setting uncertainty intervals, the excessive rejection rate is 10.6%; after introducing the uncertainty interval condition triggering mechanism, the excessive rejection rate decreases to 7.4%, while the insecure output rate remains at 5.9%. This result shows that the uncertainty interval condition triggering mechanism can avoid directly rejecting or intercepting boundary samples, thereby alleviating the excessive rejection problem, while maintaining a low insecure output rate. This indicates that the key secure attention head screening mechanism and the uncertainty interval condition triggering mechanism of this application can jointly improve the stability and usability of security control.

[0122] VI. Verification Conclusion Based on the experimental verification and comparative results above, the method described in this application reduces the insecure output rate compared to the original secure alignment model, input / output text filtering method, and fixed high-risk attention head shielding method. Furthermore, compared to schemes that only use key attention heads for joint discrimination and direct gating, the method in this application reduces the excessive rejection rate through an uncertainty interval conditional triggering mechanism. Therefore, the large-model security control method based on attention head state constraints provided in this application embodiment can achieve local, conditionally triggered security control during the inference phase by utilizing the internal state of the target secure attention head without retraining the main parameters of the large model, thereby improving security while maintaining the availability of normal requests.

[0123] Example 2 The above combination Figure 1 This application provides a detailed description of a large-model security control method based on attention head state constraints, as illustrated in its embodiments. The following, in conjunction with... Figure 2 This document describes in detail a large-model security control system based on attention head state constraints, which is used to implement the large-model security control method based on attention head state constraints provided in the embodiments of this application. Figure 2 This is a schematic diagram of a large-model safety control system based on attention head state constraints provided in an embodiment of the present invention; see reference. Figure 2 The system includes: The sample pair construction module 100 is used to construct a secure-insecure sample pair dataset; wherein, the dataset includes multiple sets of secure response samples and insecure response samples; each set of secure-insecure sample pairs shares the same input prompt and corresponds to a secure response and an insecure response, respectively; The state extraction module 200 is used to input safe response samples and unsafe response samples into the large model respectively, extract the safe state vector and unsafe state vector of the safe response samples and unsafe response samples at a specified position of each attention head; and calculate the state separation degree of each attention head based on the safe state vector and unsafe state vector. The key head filtering module 300 is used to calculate a multi-dimensional filtering index for each attention head based on the safe state vector, the unsafe state vector, and the state separation degree; and to evaluate each attention head based on the multi-dimensional filtering index to obtain a set of target safe attention heads. The discriminator training module 400 trains a pre-constructed integrated joint state discriminator based on the safe state vector and unsafe state vector of each attention head in the target safe attention head set; wherein, the integrated joint state discriminator is used to jointly discriminate the safe state of all attention heads in the target safe attention head set, and output the corresponding safe probability value and uncertainty estimate value for each safe-unsafe sample pair; The model inference real-time gating module 500 is used during the inference phase of the large model to perform joint discrimination on the safety status of all attention heads in the target safety attention head set for the current input prompt. Based on the corresponding safety probability value and uncertainty estimate value of the target input prompt output by the integrated joint state discriminator, the large model is subjected to real-time safety control.

[0124] In one feasible implementation, the sample pair construction module 100 is specifically used to: collect multiple sets of input prompts; generate safe and insecure responses for each set of input prompts; wherein the safe responses are obtained by inputting the input prompts into a large model and applying safe alignment constraints, and the insecure responses are obtained by inputting the input prompts into a large model and removing or weakening the safe alignment constraints; and construct a safe-insecure sample pair dataset using the generated multiple sets of safe and insecure responses.

[0125] In one feasible implementation, the state extraction module 200 calculates the state separation degree of each attention head based on the safe state vector and the insecure state vector using the following formula: D i = cos(h i^safe , r safe ) - cos(h i^unsafe , r safe ); in, D i Let be the state separation degree of the i-th attention head; h i^safe The security state vector of the security response sample at the specified position of the i-th attention head; h i^unsafe The unsafe state vector of the unsafe response sample at the specified position of the i-th attention head; r safe This is a preset safety reference direction vector.

[0126] In one feasible implementation, the multidimensional screening index includes at least three of the following: discriminative power index, stability index, sensitivity index, coverage index, independence index, and efficiency index. The discriminative power index measures the attention head's ability to distinguish between safe and unsafe states; a higher discriminative power index value indicates a stronger ability to distinguish between safe and unsafe states. The stability index measures the consistency of the attention head's state separation under different input prompts; a higher stability index value indicates more consistent safe discrimination behavior under different inputs. The sensitivity index measures the attention head's sensitivity to unsafe inputs. The response sensitivity index is as follows: a higher sensitivity index value indicates that the attention head is more sensitive to insecure inputs; the coverage index is used to measure the detection coverage of the attention head for different types of insecure content; a higher coverage index value indicates that the attention head has the ability to detect more types of insecure content; the independence index is used to measure the degree of information redundancy between the attention head and other selected attention heads; a higher independence index value indicates that the attention head can provide more independent information; the efficiency index is used to measure the impact of introducing the attention head on the model inference latency; a higher efficiency index value indicates that the computational cost of introducing the attention head is lower.

[0127] In one feasible implementation, the keyhead selection module 300 is specifically used for: normalizing the multidimensional selection indicators to obtain normalized scores for the multidimensional selection indicators; calculating a comprehensive score for each attention head by weighted summation based on the normalized scores of the multidimensional selection indicators and the weight coefficients of each multidimensional selection indicator; sorting all attention heads from high to low according to the comprehensive scores, and selecting the top K attention heads to form a target security attention head set; where K is a positive integer, K≤½H; and H is the number of attention heads in the large model.

[0128] In one feasible implementation, the discriminator training module 400 is specifically used to: for each safe-unsafe sample pair in the dataset, extract the safe state vector and unsafe state vector of all attention heads in the target safe attention head set, and concatenate them into a joint feature vector; Using the joint feature vector as input and the safety labels of safe-unsafe sample pairs as training targets, a pre-constructed ensemble-level joint state discriminator is trained using a binary cross-entropy loss function. The ensemble-level joint state discriminator employs a multilayer perceptron structure, including an input layer, at least one hidden layer, and an output layer. Each hidden layer uses ReLU activation and Dropout regularization. During large-scale model inference, the Dropout layer remains active. T forward propagations are performed on the same input, and the mean of the T prediction results is calculated as the safety probability value output by the output layer. The standard deviation of the T prediction results is also calculated as the uncertainty estimate of the output layer.

[0129] In one feasible implementation, the discriminator training module 400 is further configured to: perform a causal intervention experiment on each attention head in the target security attention head set to verify the causal influence of each attention head on the security output; select a set of input prompts known to produce an unsafe response, replace the internal state of the attention head with the mean of the security state vector during the large model inference process, and determine whether the output of the large model is a safe response; if so, determine that the attention head has passed the verification of the causal intervention experiment; otherwise, determine that the attention head has not passed the verification of the causal intervention experiment, and remove the attention head that has not passed the verification of the causal intervention experiment from the target security attention head set; based on the updated target security attention head set, re-execute the training operation of the integrated joint state discriminator until all security-unsafe sample pairs in the security-unsafe sample pair dataset are traversed.

[0130] In one feasible implementation, the real-time gating module 500 for model inference is specifically used for: If p safe ≥ T safe And u ≤ U threshold If the current generation state of the large language model is deemed safe, the model is allowed to continue generating normally. If p safe <T danger And u≤U threshold If the current generation state of the large language model is deemed dangerous, a large model security control operation is immediately triggered; wherein, the security control operation includes output interception, output redirection, or progressive degradation; If T danger ≤p safe <T safe , or u>U threshold Then determine the current generation of the large language model. The status is uncertain, triggering an enhanced verification process; the enhanced verification process includes: performing a secondary security check on the currently input target prompt; Where, p safe The integrated-level joint state discriminator outputs a corresponding safety probability value for the target input; u outputs a corresponding uncertainty estimate for the target input; T safe The preset safety threshold; T danger The preset danger threshold; U threshold This is a preset uncertainty threshold.

[0131] In one feasible implementation, the model inference real-time gating module 500 is further configured to: continuously collect the discrimination results of the integrated-level joint state discriminator and user feedback data during the inference phase of the large model; and periodically update the parameters of the target security attention head set and the integrated-level joint state discriminator based on the discrimination results and user feedback data.

[0132] The specific implementation method and technical effects of this system are described in the aforementioned large model security control method based on attention head state constraints, and will not be repeated here.

[0133] Example 3

[0134] This invention also provides a computer electronic device. Figure 3 This is a structural diagram of an electronic device according to an embodiment of the present invention, such as... Figure 3 As shown, this computer electronic device includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 302 or programs loaded from storage section 308 into random access memory (RAM) 303. The RAM 303 also stores various programs and data required for system operation. The CPU 301, ROM 302, and RAM 303 are interconnected via bus 304. An input / output (I / O) interface 305 is also connected to bus 304. The following components are connected to the I / O interface 305: an input section 306 including a keyboard, mouse, etc.; an output section 307 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card, modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A driver 310 is also connected to the I / O interface 305 as needed. Removable media 311, such as disks, optical discs, magneto-optical discs, semiconductor memories, etc., are installed on drive 310 as needed so that computer programs read from them can be installed into storage section 308 as needed.

[0135] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0136] Example 4

[0137] This invention also provides a computer-readable storage medium, which may be the computer-readable storage medium included in the large model security control system based on attention head state constraints described in the above embodiments; or it may be a standalone computer-readable storage medium not assembled into an electronic device. The computer-readable storage medium stores one or more programs, which are used by one or more processors to execute the large model security control method based on attention head state constraints described in this invention.

[0138] Example 5

[0139] This invention also provides a computer program product, which includes a computer program that, when executed by a computer device, implements a large model security control method based on attention head state constraints according to this invention.

[0140] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention is not limited to the specific combination of the above-described technical features, but also includes other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in this invention.

Claims

1. A large-scale model security control method based on attention head state constraints, characterized in that, include: Construct a secure-insecure sample pair dataset; wherein the dataset includes multiple sets of secure response samples and insecure response samples; each set of secure-insecure sample pairs shares the same input prompt and corresponds to secure response and insecure response respectively; Input safe response samples and unsafe response samples into the large model respectively, and extract the safe state vector and unsafe state vector of the safe response sample and the unsafe state vector at the specified position of each attention head; calculate the state separation degree of each attention head based on the safe state vector and the unsafe state vector. Based on the safe state vector, the unsafe state vector, and the state separation degree, a multidimensional screening index is calculated for each attention head; each attention head is evaluated based on the multidimensional screening index, and a target safe attention head set is obtained. Based on the safe state vector and unsafe state vector of each attention head in the target security attention head set, a pre-constructed ensemble-level joint state discriminator is trained; wherein, the ensemble-level joint state discriminator is used to jointly discriminate the security state of all attention heads in the target security attention head set, and output the corresponding security probability value and uncertainty estimate value for each pair of safe-unsafe samples. During the inference phase of the large model, the trained ensemble-level joint state discriminator is used to jointly discriminate the safety status of all attention heads in the target safety attention head set for the current input target prompt. Based on the corresponding safety probability value and uncertainty estimate value of the target input prompt output by the ensemble-level joint state discriminator, the large model is subjected to real-time safety control.

2. The large model security control method based on attention head state constraints as described in claim 1, characterized in that, Construct a secure-insecure sample pair dataset, including: Collect multiple sets of input prompts; generate safe and insecure responses for each set of input prompts; wherein, the safe response is obtained by inputting the input prompt into a large model and applying a safe alignment constraint, and the insecure response is obtained by inputting the input prompt into a large model and removing or weakening the safe alignment constraint; construct a safe-insecure sample pair dataset using the generated multiple sets of safe and insecure responses.

3. The large model security control method based on attention head state constraints as described in claim 2, characterized in that, The formula for calculating the state separation degree of each attention head based on the safe state vector and the unsafe state vector is as follows: in, D i Let be the state separation degree of the i-th attention head; h i^safe The security state vector of the security response sample at the specified position of the i-th attention head; h i^unsafe The unsafe state vector of the unsafe response sample at the specified position of the i-th attention head; r safe This is a preset safety reference direction vector.

4. The large model security control method based on attention head state constraints as described in claim 3, characterized in that, The multidimensional screening metrics include at least three of the following: discriminative power, stability, sensitivity, coverage, independence, and efficiency. The discriminative power metric measures the attention head's ability to distinguish between safe and unsafe states; a higher discriminative power value indicates a stronger ability to differentiate between safe and unsafe states. The stability metric measures the consistency of the attention head's state separation under different input cues; a higher stability metric value indicates more consistent safe discrimination behavior under different inputs. The sensitivity metric measures the attention head's responsiveness to unsafe inputs. The higher the sensitivity index value, the more sensitive the attention head is to insecure inputs; the coverage index measures the detection range of different types of insecure content by the attention head, and the higher the coverage index value, the more types of insecure content the attention head can detect; the independence index measures the degree of information redundancy between the attention head and other selected attention heads, and the higher the independence index value, the more independent information the attention head can provide; the efficiency index measures the impact of introducing the attention head on the model inference latency, and the higher the efficiency index value, the lower the computational cost of introducing the attention head.

5. The large model security control method based on attention head state constraints as described in claim 4, characterized in that, Each attention head is evaluated based on the aforementioned multidimensional screening metrics, resulting in a target set of secure attention heads, including: The multidimensional screening indicators are normalized to obtain the normalized scores of the multidimensional screening indicators. Based on the normalized scores of the multidimensional screening indicators and the weight coefficients of each multidimensional screening indicator, a comprehensive score for each attention point is obtained by weighted summation. All attention heads are sorted from highest to lowest according to the comprehensive score, and the top K attention heads are selected to form the target safe attention head set; where K is a positive integer, K≤½H; where H is the number of attention heads in the large model.

6. The large model security control method based on attention head state constraints as described in claim 5, characterized in that, Based on the secure state vectors and insecure state vectors of each attention head in the target secure attention head set, a pre-constructed ensemble-level joint state discriminator is trained, including: For each safe-unsafe sample pair in the dataset, extract the safe state vector and unsafe state vector of all attention heads in the target safe attention head set, and concatenate them into a joint feature vector; Using the joint feature vector as input and the safety labels of safe-unsafe sample pairs as training targets, a pre-constructed ensemble-level joint state discriminator is trained using a binary cross-entropy loss function. The ensemble-level joint state discriminator employs a multilayer perceptron structure, including an input layer, at least one hidden layer, and an output layer. Each hidden layer uses ReLU activation and Dropout regularization. During large-scale model inference, the Dropout layer remains active. T forward propagations are performed on the same input, and the mean of the T prediction results is calculated as the safety probability value output by the output layer. The standard deviation of the T prediction results is also calculated as the uncertainty estimate of the output layer.

7. The large model security control method based on attention head state constraints as described in claim 6, characterized in that, After the integrated joint state discriminator is trained, the following is also included: A causal intervention experiment is performed on each attention head in the target security attention head set to verify the causal impact of each attention head on the security output; a set of input cues known to produce unsafe responses are selected, and during the large model inference process, the internal state of the attention head is replaced with the mean of the security state vector, and it is determined whether the output of the large model is a safe response; if so, the attention head is determined to have passed the causal intervention experiment verification; otherwise, the attention head is determined to have failed the causal intervention experiment verification, and the attention head that has failed the causal intervention experiment verification is removed from the target security attention head set. Based on the updated target security attention head set, the training operation of the integrated joint state discriminator is re-executed until all safe-unsafe sample pairs in the safe-unsafe sample pair dataset are traversed.

8. The large model security control method based on attention head state constraints as described in claim 7, characterized in that, Based on the target input prompts output by the integrated joint state discriminator, corresponding safety probability values ​​and uncertainty estimates are used to perform real-time safety control on the large model, including: If p safe ≥ T safe And u ≤ U threshold If the current generation state of the large language model is deemed safe, the model is allowed to continue generating normally. If p safe <T danger And u≤U threshold If the current generation state of the large language model is deemed dangerous, a large model security control operation is immediately triggered; wherein, the security control operation includes output interception, output redirection, or progressive degradation; If T danger ≤p safe <T safe , or u>U threshold If the current generation state of the large language model is determined to be uncertain, the enhanced verification process is triggered; the enhanced verification process includes: performing a secondary security check on the target input prompt of the current input; Where, p safe The integrated-level joint state discriminator outputs a corresponding safety probability value for the target input; u outputs a corresponding uncertainty estimate for the target input; T safe The preset safety threshold; T danger The preset danger threshold; U threshold This is a preset uncertainty threshold.

9. The large model security control method based on attention head state constraints as described in any one of claims 1 to 8, characterized in that, Also includes: During the inference phase of the large model, the discrimination results of the integrated joint state discriminator and user feedback data are continuously collected; the parameters of the target security attention head set and the integrated joint state discriminator are updated periodically based on the discrimination results and user feedback data.

10. A large-scale model safety control system based on attention head state constraints, characterized in that, include: The sample pair construction module is used to construct a secure-insecure sample pair dataset; wherein, the dataset includes multiple sets of secure response samples and insecure response samples; each set of secure-insecure sample pairs shares the same input prompt and corresponds to a secure response and an insecure response, respectively; The state extraction module is used to input safe response samples and unsafe response samples into the large model respectively, extract the safe state vector and unsafe state vector of the safe response sample and the unsafe state vector at a specified position of each attention head; and calculate the state separation degree of each attention head based on the safe state vector and the unsafe state vector. The key head filtering module is used to calculate a multi-dimensional filtering index for each attention head based on the safe state vector, the unsafe state vector, and the state separation degree; and to evaluate each attention head based on the multi-dimensional filtering index to obtain a set of target safe attention heads. The discriminator training module trains a pre-constructed integrated joint state discriminator based on the safe state vector and unsafe state vector of each attention head in the target security attention head set. The integrated joint state discriminator is used to jointly discriminate the security state of all attention heads in the target security attention head set and output the corresponding security probability value and uncertainty estimate value for each safe-unsafe sample pair. The real-time gating module for model inference is used during the inference phase of the large model to perform joint discrimination on the safety status of all attention heads in the target safety attention head set for the current input prompt. Based on the corresponding safety probability value and uncertainty estimate value of the target input prompt output by the joint state discriminator, the large model is subjected to real-time safety control.

Images