Large model security alignment method bridging harmful belief-refusal behavior

Abstract

The application discloses a harmful belief-refusal behavior bridging large model security alignment method, and relates to the technical field of artificial intelligence large language model security alignment. In view of the problems of single refusal signal channel and excessive refusal of benign prompt of an existing method, the method extracts hidden states at the end of the instruction token position and the start of the response token position on the basis of freezing the weight of the large language model, calculates harmfulness direction and four centroids of harmfulness-benignness and refusal-acceptance as training anchor points, defines belief scores, bridging scores and refusal scores, jointly optimizes bridging loss, belief loss, gated refusal loss, benign KL loss and language modeling loss, and merges the adapter back to the basic model for deployment after training. The method makes the refusal behavior simultaneously dominated by two independent channels of refusal direction and harmfulness belief, realizes robust defense against various jailbreak attacks under the premise of not depending on large-scale alignment data, keeps a low refusal rate of benign prompt, and retains general ability.

Description

Technical Field

[0001] This invention relates to the field of secure alignment technology for large language models in artificial intelligence, and in particular to a secure alignment method for large models that bridges harmful beliefs and refusal to answer behaviors. Background Technology

[0002] Large Language Models (LLMs) have been widely used in dialogue generation, code assistance, and question answering. To prevent them from generating harmful content under malicious prompts, the industry generally performs security alignment on pre-trained models, causing the model to output a rejection when faced with harmful requests. However, even with security alignment, the model can still be induced to produce insecure outputs under various jailbreak attacks, reflecting structural deficiencies in the internal representation level of existing alignment methods.

[0003] Existing defenses against jailbreak attacks can be broadly categorized into two types: one is behavior-level supervision-based methods, including Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO), and supervised fine-tuning based on rejection templates. These methods primarily strengthen rejection patterns at the output token sequence level. A Chinese patent (CN 118349643 A) discloses a method for fine-tuning a pre-trained large language model using Low-Rank Adaptation (LoRA) based on a secure corpus, merging the target LoRA weights into the model to improve its security. However, this approach only supervises at the output behavior level, making the rejection signal easily bypassable in scenarios such as prefix padding attacks and decoding-side attacks. The other type is representation space intervention-based methods, including activation guidance, rejection direction ablation, and circuit rerouting. These methods enhance the robustness of internal representations by intervening in the direction or subspace of the model's hidden states during training or inference. Chinese patent CN 121211439 A discloses a method that uses a consistency loss function to guide the intermediate representation of a malicious request to move closer to the intermediate representation of a compliant request, thereby interfering with the model's security alignment mechanism. However, this approach typically only strengthens a single representation channel, offering limited robustness against adversarial attacks targeting that channel, and often results in over-rejection of benign suggestions. Another Chinese patent CN 121457619 A discloses a two-layer defense framework: a dynamic malicious keyword library on the input side, model training on the model side to learn feedback information, and an output review library on the output side. However, this approach fails to fully utilize the structural characteristics of the model's internal representations; specifically, there is a lack of explicit connection between the harmful belief encoded at the end of the instruction and the rejection behavior encoded at the beginning of the response, causing the model to output a compliant response even when it can recognize the harmfulness of the request under attack.

[0004] Therefore, there is an urgent need to propose an alignment method that can explicitly connect harmful beliefs and refusal behaviors during the training phase, so that the refusal decision is supported by multiple internal representation channels at the same time, in order to improve the robustness of the model to jailbreak attacks without significantly increasing the training cost or impairing benign capabilities. Summary of the Invention

[0005] The purpose of this invention is to provide a large-scale model secure alignment method that bridges harmful beliefs and rejection behaviors, in order to solve the following technical problems existing in the prior art: Existing large-scale language model secure alignment methods lack an explicit connection between the harmful belief encoded at the end of the instruction and the rejection behavior encoded at the beginning of the response, resulting in the model being able to identify the harmfulness of the request under jailbreak attacks but still outputting a compliant response; the rejection signal channel is single, lacking robustness in scenarios such as rejection direction ablation, prefix padding, and code attacks; and over-rejection often occurs in benign prompts, impairing the usefulness of the model.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A large-scale model security alignment method for bridging harmful beliefs and refusal to respond, executed by a computing device including a processor and memory, includes the following steps: constructing an alignment training dataset, obtaining harmful and benign cues and their harmful labels; classifying the responses to the cues based on a frozen underlying large language model; identifying inconsistent samples where the behavior of the underlying model is inconsistent with the labels, the inconsistent samples including samples where the underlying model responds to harmful cues with acceptance and samples where the underlying model responds to benign cues with refusal to respond; estimating frozen anchor points, on a preset set of frozen underlying large language models, respectively at the end of user commands. The hidden state is extracted from the end-word position and the response start-word position. The processor calculates the difference between the mean values ​​of harmful and benign prompts at the end-word position of the instruction as the harmful direction. The mean values ​​of harmful and benign prompts at the end-word position of the instruction, as well as the mean values ​​of rejection and acceptance prompts at the response start-word position, are calculated as four classification centroids. The harmful direction and the four classification centroids are frozen during subsequent training. A low-rank adapter is constructed and a representation-level score is defined. The low-rank adapter is attached to the attention and feedforward projection of the frozen basic large language model. At each preset layer, a signal level is defined. The reading score is defined as the difference in cosine similarity between the hidden state of the last word of the instruction and the two centroids of harmful and benign responses. The bridging score is defined as the cosine similarity between the hidden state of the first word of the response and the harmful direction. The rejection score is defined as the difference in cosine similarity between the hidden state of the first word of the response and the two centroids of rejection and acceptance. An arithmetic average is calculated for the scores within a preset layer set. The bridging loss is calculated by the processor, which passes the bridging score through a sigmoid function and then calculates a weighted binary cross-entropy with the harmful label. The weighting increases the sample weight of inconsistent samples by a preset multiple to more strongly correct the base model. The system identifies behaviors inconsistent with the label at the initial response position; calculates the belief loss by having the processor pass the belief score through a sigmoid function and then calculate a weighted binary cross-entropy with the harmful label, where the weighting increases the sample weight of inconsistent samples by a preset factor; and calculates the gated rejection loss by constructing a sample-level gate based on the belief score. The gate is equal to applying a stopping gradient operator to the belief score, multiplying it by a sign consistent with the label, then amplifying it with a scaling factor and taking the sigmoid function, such that the gate is greater than 0.9 on samples where the belief and label are consistent, and less than 0 on samples where the belief and label are clearly conflicting.1. Multiply the gating values ​​sample by sample by the binary cross-entropy corresponding to the rejection score to obtain the gating rejection loss; calculate the benign KL loss and language modeling loss. On a subset consisting only of benign prompts, calculate the KL divergence of the probability distribution of the next word in the frozen basic large language model and the model with the low-rank adapter attached as the benign KL loss. For samples whose behavior in the basic model is consistent with the label, calculate the cross-entropy of the next word in the target response obtained by greedily decoding the frozen basic large language model as the language modeling loss; perform joint optimization by linearly weighting the bridging loss, belief loss, gating rejection loss, benign KL loss and language modeling loss according to preset non-negative weights to obtain the total loss, and perform joint optimization only on the parameters of the low-rank adapter; merge and deploy the adapters. After training, merge the low-rank adapters back into the frozen basic large language model and deploy them with the same number of parameters and inference latency as the frozen basic large language model. The "frozen basic large language model" refers to a basic large language model whose parameters remain unchanged during training. The lexical units are tokens. The preset layer set is selected from the intermediate layers of the frozen basic large language model. The low-rank adapter has a rank of 8 to 32, a scaling factor of 1 to 3 times its rank, a dropout rate of 0 to 0.2, a preset multiplier of a positive real number greater than 1, and a scaling factor of a positive real number. The greedy decoding refers to selecting the lexical unit with the highest probability as the output in each generation step. The preset non-negative weights include bridging loss weights, belief loss weights, gating rejection loss weights, benign KL loss weights, and language modeling loss weights.

[0007] Through the above technical solution, this invention estimates the harmfulness direction and four classification centroids as training anchors on a frozen basic large language model, and jointly optimizes bridging loss, belief loss, gated rejection loss, benign KL loss, and language modeling loss on a low-rank adapter. It explicitly transmits the harmfulness belief already present in the basic model at the end of the instruction to the response start position, so that the rejection behavior is simultaneously governed by two independent channels: the normal rejection direction and the harmfulness belief. Thus, without relying on large-scale preference alignment data or modifying the reasoning structure of the basic model, it has significant robustness to various jailbreak attacks, maintains a low rejection rate for benign prompts, and retains general capabilities.

[0008] Furthermore, the sample weight of the inconsistent sample is a preset multiple greater than 1 of that of the non-inconsistent sample, preferably 2 to 5 times. By assigning higher sample weights to inconsistent samples, the training more effectively corrects the problem of the base model's behavior being inconsistent with the label on these samples, thereby improving the alignment effect.

[0009] Furthermore, the position of the last word element of the instruction is the position of the last word element of the user instruction in the dialogue template, and the position of the first word element of the response is the position of the boundary word element before the first word element of the assistant's response in the dialogue template. For large language models of different dialogue templates, the positions of the last word element of the instruction and the first word element of the response are automatically located according to the special word elements corresponding to the template. This automatic location mechanism enables the method of the present invention to adapt to various dialogue templates, such as <|eot_id|> and <|end_header_id|> in Llama-3, and Gemma's...<end_of_turn> and<start_of_turn> Models, Mistral's [ / INST] and [INST], etc., improve the generality of the method.

[0010] Furthermore, the estimation of the harmful direction employs the difference-means method: at the end-word position of the instruction, the hidden states of several harmful prompts are averaged, and the hidden states of several benign prompts are averaged. The difference between these two values, normalized to a unit vector, is the harmful direction. The difference-means calculation and the estimation of the four classification centroids are performed only once on the frozen basic large language model. This difference-means method is simple and efficient, obtaining stable harmful directions without iterative optimization. Moreover, the anchor points remain frozen during training, avoiding backpropagation perturbations and improving training stability.

[0011] Preferably, the bridging loss is expressed as follows: the bridging loss equals the sum of the binary cross-entropy between the sample weights multiplied by the probability obtained by applying the bridging score using the Sigmoid function and the harmful label on all training samples, divided by the sum of the sample weights; wherein the sample weights are a preset multiple greater than 1 on inconsistent samples and 1 on the remaining samples. The bridging loss explicitly establishes a bridging channel from the end of the instruction to the beginning of the response by pushing the hidden state at the response initiation position towards harmfulness, thus making the rejection decision governed by harmfulness belief signals.

[0012] Preferably, the expression for the belief loss is: in, The normalization constant for sample weights. ω(x) is the sample weight. BCE(p,y)=-y·log(p)-(1-y)·log(1-p) is the binary cross-entropy function. σ(·) is the Sigmoid function. τ is the temperature coefficient. B(x) represents the belief score. y(x)∈{0,1} is the harmfulness label of sample x. D represents the complete training dataset.

[0013] Preferably, the gating expression is: Where σ(·) is the Sigmoid function, α is the scaling coefficient and a positive real number ranging from 2 to 8, y is the harmfulness label of sample x and y∈{0,1}, (2y-1) takes +1 when the label is harmful and -1 when it is benign, and sg[·] is the stopping gradient operator so that the backpropagation of the gating does not affect the hidden state corresponding to the belief score B(x), where B(x) is the belief score. The gating mechanism enables training to train the rejection direction at full intensity only on clear samples where the belief and label are consistent, avoiding false reinforcement of rejection on samples where the belief and label conflict, thereby significantly reducing the over-rejection rate.

[0014] Preferably, the benign KL loss is calculated only on the benign cues subset, and no KL constraint is imposed on the harmful cues subset, to avoid retaining undesirable acceptance behavior of the base large language model on harmful cues. This selective constraint design ensures that the model maintains behavior close to the base model on benign cues, while fully correcting to rejection behavior on harmful cues.

[0015] Preferably, the language modeling loss uses the greedy decoding result of the frozen base large language model for benign prompts as a soft objective. For inconsistent samples, the sample weights of the language modeling loss are reset to zero, preventing the language modeling loss from inversely offsetting the corrective effects of the bridging loss, belief loss, and gating rejection loss on the inconsistent samples. This zero-weight strategy ensures that the language modeling loss only works on consistent samples, maintaining the model's language generation capability without interfering with the correction of inconsistent samples.

[0016] Preferably, the preset layer set is selected from the intermediate layers of the large language model. For the Llama-3 and Mistral series, layers 12, 16, 20, and 24 are used; for the Gemma series, layers 10, 14, 18, and 22 are used. For other series of models, four intermediate layers are selected proportionally to the number of layers, so that the relative position of the selected layer in the total number of layers of the model corresponds to the relative position of the selected layer in the reference model. The intermediate layers have the highest separability for harmful signals. Signals in earlier layers have not yet stabilized, and signals in later layers have been processed for tasks, both of which are not conducive to bridging training.

[0017] Preferably, the trainable modules of the low-rank adapter include attention queries, keys, values, output projections, and gates, up projections, and down projections in the feedforward network; the rank of the low-rank adapter is 8 to 32, the scaling factor is twice the rank, and the dropout rate is 0 to 0.1. These parameters are set to ensure training efficiency while providing sufficient capacity to support the joint optimization of the five losses.

[0018] Preferably, the preset non-negative weights include: a bridging loss weight of 0.5 to 2.0, a belief loss weight of 0.5 to 2.0, a gating rejection loss weight of 0.5 to 2.0, a benign KL loss weight of 0.1 to 0.5, and a language modeling loss weight of 1.0. These weight configurations ensure that the five losses are balanced in the joint optimization, effectively correcting inconsistent samples while maintaining benign capabilities and language generation capabilities.

[0019] Compared with the prior art, the present invention has the following beneficial effects: 1. By directly supervising harmful beliefs at the end of the instruction and explicitly establishing a bridging channel from the initial position of the response to the rejection position, the rejection decision is simultaneously governed by two independent channels: the normal rejection direction and the harmful belief. Compared with methods that only strengthen the rejection direction, the robustness against jailbreak attacks such as rejection direction ablation, prefix filling, and code attacks is significantly improved. On Llama-3-8B-Instruct, the attack success rates of RFA, Prefill, and CodeAttack attacks decreased from 79.4%, 67.8%, and 87.1% to 0.6%, 0.0%, and 0.0%, respectively, with relative reductions of approximately 99%, 100%, and 100%, achieving the lowest or tied lowest attack success rates across all eight attack families.

[0020] 2. By adjusting the rejection loss through sample-level gating based on belief scoring, the rejection direction is trained with full intensity only on clear samples where the belief and label are consistent. Compared with the method of uniformly reinforcing rejection for all samples, this method can significantly reduce the success rate of harmful prompt attacks while maintaining a low over-rejection rate for benign prompts. On Llama-3-8B-Instruct, the XSTest compliance rate is improved from 95.2% to 95.6% of the base model without introducing significant over-rejection. On Mistral-7B-Instruct-v0.2, the XSTest compliance rate is 92.0%, which is only 0.4 percentage points lower than the base model's 92.4%, significantly better than the performance of existing methods that reduce the XSTest compliance rate by 7 to 12 percentage points on Mistral.

[0021] 3. The design of frozen anchor points ensures that the harmful direction and the four classification centroids are not disturbed by backpropagation during training, so that the bridging channel is always aligned with the harmful subspace initially represented by the base model, improving training stability and cross-model transferability. Consistent defense effects were achieved on three different architectures: Llama-3, Gemma, and Mistral, verifying the versatility of the method.

[0022] 4. The method uses a low-rank adapter for training. After training, the adapter is merged back into the basic large language model. The deployed model maintains consistency with the original model in terms of parameter count and inference latency. There is no need to modify the network structure and inference process of the basic model. It has strong engineering deployability. After training on Llama-3-8B-Instruct, the merged and deployed model remains robust to the basic model in general performance indicators such as MMLU and MT-Bench. The MMLU decreased from 64.5% to 64.2% of the basic model, with a difference of no more than 1 percentage point. The MT-Bench decreased from 7.07 to 6.38.

[0023] 5. The method does not rely on large-scale preference alignment data. The training data is relatively small and can be constructed from existing public datasets. Training can be completed in a few hours with a single A800-80GB GPU. The computational requirements are low. In this embodiment, the training dataset size is 3208 pairs, and the training time is about 3.5 to 4.2 GPU hours. Compared with reinforcement learning methods that require tens of thousands to hundreds of thousands of preference pairs and tens to hundreds of GPU hours, the training cost is significantly reduced, and it can be universally transferred to various open-source models. Attached Figure Description

[0024] Figure 1 A general framework diagram for safe alignment of a large language model based on harmful beliefs-refusal behavior bridging.

[0025] Figure 2 Flowchart of a large-scale model for safe alignment of harmful beliefs and refusal-to-respond behavior.

[0026] Figure 3 This is a semantic diagram showing the position of the last word in a command and the position of the first word in a response to a dialog template.

[0027] Figure 4 This is a block diagram of the architecture of a secure alignment system for a large language model based on harmful beliefs and refusal to respond behaviors. Detailed Implementation

[0028] The present invention will now be described in further detail with reference to the accompanying drawings.

[0029] This invention provides a large-scale model secure alignment method for bridging harmful beliefs and rejection behaviors. Its core improvement lies in estimating the harmful direction and four classification centroids as training anchors on a preset set of layers of a frozen basic large language model. Furthermore, it jointly optimizes the bridging loss, belief loss, gated rejection loss, benign KL loss, and language modeling loss on a model with a low-rank adapter. This explicitly transmits the harmful beliefs already present in the basic model at the end of the instruction to the beginning of the response, ensuring that rejection behavior is simultaneously governed by two independent channels: the normal rejection direction and the harmful belief. Therefore, without relying on large-scale preference alignment data or modifying the reasoning structure of the basic model, it exhibits significant robustness against various jailbreak attacks, maintains a low rejection rate for benign prompts, and retains general capabilities.

[0030] Example 1 like Figure 1 and Figure 2 As shown, this embodiment provides a secure alignment method based on the Llama-3-8B-Instruct large language model.

[0031] This method achieves robust defense against jailbreak attacks and low over / rejection of benign prompts by estimating the harmful orientation and centroid on a frozen base model and jointly optimizing five loss terms on a low-rank adapter.

[0032] I. System Configuration The hardware configuration used in this embodiment is as follows: operating system Ubuntu 22.04 (a Linux distribution operating system), processor Intel(R) Xeon(R) series CPU (central processing unit), and GPU (graphics processing unit) NVIDIA A800 80GB×1.

[0033] The software configuration used in this embodiment is as follows: the basic large language model Llama-3-8B-Instruct (an open-source large language model from Meta, with 32 decoding layers, 32 attention heads per layer, and 4096 hidden dimensions), the low-rank adapter implementation uses the PEFT / LoRA framework (a parameter-efficient fine-tuning framework), the training framework is PyTorch (a deep learning framework) + bfloat16 (a 16-bit floating-point number format), and the optimizer is AdamW (an adaptive gradient optimization algorithm); the auxiliary evaluators used for response classification and judgment are GPT-4o (a large language model developed by OpenAI) and HarmBench-Llama-2-13b-cls (a harmfulness classifier for the HarmBench benchmark).

[0034] II. Constructing the Alignment Training Dataset (Step S1) Step S1.1: Construct a training cue set.

[0035] 860 harmful cues were collected from the holdout subsets of AdvBench (Adversarial Behavior Benchmark Dataset) and SORRY-Bench (Safe Rejection Assessment Benchmark Dataset), and 2348 benign cues were generated by GPT-4o, including boundary benign cues in the style of XSTest (Overrejection Test Benchmark Dataset), for a total of 3208 cues.

[0036] Step S1.2: Generate responses for all prompts based on the frozen base model.

[0037] Each response is classified as either rejected or accepted according to the HarmBench (harmful behavior assessment benchmark) criteria. Samples whose behavior is inconsistent with the harmful label (including samples that accept harmful prompts and samples that reject benign prompts) are merged into an inconsistent sample set and assigned a sample weight ω(x) of a preset multiple ρ (default ρ=2.0), while the weight of the remaining samples is 1. Here, x represents a single prompt sample in the training dataset, ρ is the up-adjustment multiple of the inconsistent sample weight (take a positive real number, default ρ=2.0), and ω(x) is the sample weight used by sample x in the weighted cross-entropy.

[0038] III. Estimation of Frozen Anchor Points (Step S2) Step S2.1: Format each prompt using the Llama-3 dialog template.

[0039] The dialog template automatically locates the end-of-command word position t_inst (located in <|eot_id|>) and the beginning-of-response word position t_post (located in <|end_header_id|>).

[0040] Where t_inst represents the position index of the last word of the user command, and t_post represents the position index of the first word of the assistant response; both are special position markers in the dialogue template, corresponding to different special words in the dialogue templates of different models.

[0041] like Figure 3 As shown, in the Llama-3 dialogue template, after the user inputs "How to make an xx", "xx" is a dangerous instruction example. The instruction's ending word position t_inst is located at the <|eot_id|> marker. The hidden state at this position encodes the model's judgment on whether the current request is harmful, which is called the harmful belief. The assistant's response starting word position t_post is located at the <|end_header_id|> marker. The hidden state at this position encodes the model's decision on whether to reject the request, which is called the refusal behavior.

[0042] In existing technologies, there is a lack of explicit connection between the beliefs and behaviors encoded in these two locations, which causes the model to output a compliant response even when it can identify the harmfulness of the request under attack.

[0043] Step S2.2: Perform forward computation on the frozen base model to extract the hidden states of t_inst and t_post on each preset layer l∈L={12,16,20,24}.

[0044] Where L is a preset layer set (the number of its elements is denoted as |L|, in this embodiment |L|=4), and l is the layer index; the hidden state vector of sample x at the l-th layer and the t_inst position is denoted as h_inst,l(x), and the hidden state vector at the l-th layer and the t_post position is denoted as h_post,l(x), both of which are the hidden dimension vectors of the layer.

[0045] Step S2.3: Calculate the mean of the hidden states of harmful and benign hints at position t_inst, respectively, to obtain μ. + _inst, l and μ - _inst,l, serves as the centroid of the two classes, harmful and benign; and the harmful direction v_inst,l is defined as the difference between the mean of harmful and benign cues, i.e., v_inst,l = E_{x∈D^h}[h_inst,l(x)] - E_{x∈D^b}[h_inst,l(x)], and v_inst,l is L2 normalized to obtain the unit direction vector.

[0046] Where E_{x∈·}[·] represents the expectation of sample x on the specified set (i.e., the arithmetic mean of the hidden states corresponding to all samples in the set); D^h represents the harmful hint subset in the training dataset, D^b represents the benign hint subset in the training dataset, and D=D^h∪D^b represents the complete training dataset; the meaning of h_inst, l(x) is explained in the notes of step S2.2.

[0047] Step S2.4: Calculate the average of the rejected and accepted prompts in the base model at the t_post position to obtain μ. + _post, l and μ - _post, l, as the centroid of two types: rejection and acceptance.

[0048] Step S2.5: Set v_inst, l, μ + _inst, l, μ - _inst, l, μ + _post, l, μ - Five anchor points, _post and l, are saved as frozen tensors; during subsequent training, all anchor points remain frozen and do not participate in gradient updates.

[0049] IV. Construct a low-rank adapter and define representation-level scoring (step S3) Step S3.1: Attach the LoRA adapter to the attention q / k / v / o projection (i.e., query, key, value, and output projection) of the base model and the gate / up / down projection of the feed-forward network (FFN), with rank r=16, scaling factor s_lora=32 (i.e., s_lora=2r), and dropout rate of 0.05; here, the scaling factor s_lora is only used inside the LoRA adapter and is different from the gating scaling factor α below.

[0050] Step S3.2: Define the belief score B(x), bridging score M(x), and refusal score R(x) as follows: B(x) = (1 / |L|)·Σ[cos(h_inst,l(x), μ + _inst,l) - cos(h_inst,l(x), μ - _inst,l)] (Equation 1) M(x) = (1 / |L|)·Σ cos(h_post,l(x), v_inst,l) (Formula 2) R(x) = (1 / |L|)·Σ[cos(h_post,l(x), μ + _post,l) - cos(h_post,l(x), μ - _post,l)] (Equation 3) Where |L| represents the number of elements in the preset layer set L (i.e., the cardinality, in this embodiment |L|=4); Σ_{l∈L} represents the summation of all layer indices l in the preset layer set L, and (1 / |L|)·Σ_{l∈L}(·) is the arithmetic mean of the scores of each layer in L; the meanings of h_inst,l(x) and h_post,l(x) are explained in the annotation of step S2.2, μ + _inst, l, μ - _inst, l, μ + _post, l, μ - The meanings of _post,l and v_inst,l are explained in steps S2.3 and S2.4; cos(a,b) represents the cosine similarity between vector a and vector b, i.e., cos(a,b) = a·b / (‖a‖·‖b‖), and its value range is [-1, 1].

[0051] All scores are scalars in the range [-1, 1].

[0052] V. Calculate bridging loss (step S4) Map M(x) to a probability of [0, 1] using the Sigmoid function σ(·), and calculate the binary cross-entropy BCE with the harmful label y(x) ∈ {0, 1}. Then, weight the probabilities using the sample weights ω(x). L_bridge = (1 / Z)·Σ_{x∈D} ω(x)·BCE(σ(M(x) / τ), y(x)) (Equation 4) L_bridge forces the hidden state at t_post to retain harmful signals along the v_inst direction, thereby explicitly establishing a bridging channel from t_inst to t_post; L_bel directly supervises the belief score at t_inst to ensure that the signal transmitted through the bridging channel is consistent with the label.

[0053] VI. Calculate the loss of belief (step S5) Map B(x) to a probability of [0, 1] using the Sigmoid function σ(·), and calculate the binary cross-entropy BCE with the harmful label y(x) ∈ {0, 1}. Then, weight the values ​​using the sample weights ω(x) to obtain the belief loss. L_bel = (1 / Z)·Σ_{x∈D} ω(x)·BCE(σ(B(x) / τ), y(x)), Z = Σ_{x∈D} ω(x) (Equation 5) The temperature coefficient τ is set to 1.0 by default, and can be adjusted to 0.1, 0.5, 2.0, or 5.0 through ablation design experiments; the sample weight ω(x) is set to ρ=2.0 for inconsistent samples and 1 for the rest.

[0054] Where D represents the complete training dataset (D=D^h∪D^b), σ(·) is the Sigmoid function, BCE(p,y)=-y·log(p)-(1-y)·log(1-p) is the binary cross-entropy function, y(x)∈{0,1} is the harmfulness label of sample x (1 represents harmfulness, 0 represents benignness), and Z=Σ_{x∈D}ω(x) is the sample weight normalization constant (i.e. the sum of the weights of all training samples).

[0055] L_bridge forces the hidden state at t_post to retain harmful signals along the v_inst direction, thereby explicitly establishing a bridging channel from t_inst to t_post; L_bel directly supervises the belief score at t_inst to ensure that the signal transmitted through the bridging channel is consistent with the label.

[0056] 7. Calculate the gated rejection loss (step S6) Step S6.1: Construct sample-level gating g(x, y) based on belief score B(x): g(x,y) = σ(α·(2y-1)·sg[B(x)]) (Formula 6) Where y is the harmfulness label y(x) corresponding to sample x (1 represents harmfulness, 0 represents benignness), sg[·] is the stopping gradient operator, so that the gating does not affect the hidden state corresponding to B(x) during backpropagation; α is the scaling factor, with a default value of α=4.0; (2y-1) takes +1 when the label y=1 and -1 when y=0, so that the gating is close to 1 on samples where the belief is consistent with the label and close to 0 on samples where the belief is obviously conflicting with the label.

[0057] Step S6.2: Map R(x) to a probability via σ, calculate the BCE with y, multiply by the gate g(x, y) and sample weight ω(x), and then calculate the weighted average to obtain the gated rejection loss: L_ref = (1 / Z)·Σ_{x∈D} ω(x)·g(x,y(x))·BCE(σ(R(x) / τ), y(x)) (Equation 7) The introduction of gating allows the present invention to train the rejection direction at full intensity only on clear samples where the belief-label are consistent, avoiding false reinforcement of rejection on samples where the belief and label conflict, thereby significantly reducing the over-rejection rate for benign prompts.

[0058] 8. Calculate the benign KL loss and language modeling loss (step S7) Step S7.1: Construct a benign subset D_b = {x|y(x) = 0}, and calculate the KL divergence of the model relative to the base model after adapter loading on D_b for the probability distribution of the next lexical at position t_post. L_kl = E_{x∈D_b}[KL(p_θ(·|x, t_post) ‖ p_base(·|x, t_post))] (Equation 8) The KL divergence is an information-theoretic metric that measures the difference between two probability distributions.

[0059] Where θ represents the set of trainable parameters after attaching the LoRA adapter (only the low-rank adapter parameters participate in gradient updates, and the base model parameters remain frozen); p_θ(·|x, t_post) represents the probability distribution of the next word predicted by the model after attaching the adapter at the t_post position of sample x; p_base(·|x, t_post) represents the probability distribution of the next word predicted by the base model at the same position; KL(·‖·) is the Kullback-Leibler divergence, used to measure the difference between two probability distributions.

[0060] Benign KL is computed only on a benign subset to avoid retaining undesirable acceptance behaviors of the underlying model on harmful cues.

[0061] Step S7.2: For samples whose behavior is consistent with the label in the basic model, use the next word cross-entropy as the language modeling loss L_lm according to the target response obtained by the greedy decoding of the basic model; reset the sample weights of inconsistent samples to zero to avoid L_lm from inversely canceling the corrective effect of L_bridge, L_bel, and L_ref on these samples.

[0062] IX. Joint Optimization (Step S8) Construct the total loss L_total (Note: The subscript "total" here is used to distinguish it from L, which represents the preset set of layers): L_total = L_lm + λ_bel·L_bel + λ_ref·L_ref + λ_bridge·L_bridge +λ_kl·L_kl (Equation 9) Where L_total represents the total loss (the subscript total is used as an identifier to distinguish it from L, which represents the set of preset layers); L_lm is the language modeling loss, L_bel is the belief loss, L_ref is the gated rejection loss, L_bridge is the bridging loss, and L_kl is the benign KL loss; λ_bel, λ_ref, λ_bridge, and λ_kl are the non-negative weighted coefficients of the belief loss, gated rejection loss, bridging loss, and benign KL loss, respectively (the language modeling loss coefficient is fixed at 1.0).

[0063] In this embodiment, λ_bel=λ_ref=λ_bridge=1.0, λ_kl=0.2; the optimizer is AdamW, the learning rate is 2e-4, the batch size is 32 (batch=1×gradient accumulation 32), the training is carried out for 3 epochs, bfloat16, and the training time for a single A800-80GB card is about 3.5-4.2 GPU-hours.

[0064] 10. Adapter Consolidation Deployment (Step S9) After training, the LoRA adapter weights are merged back into the base model weights. The merged model is completely identical to the base model in terms of parameter count, memory usage, and inference latency, and can be deployed without any modifications to the inference code.

[0065] XI. Performance Verification This embodiment evaluates the attack success rate (ASR) of eight attack families on 159 non-copyright behaviors of the HarmBench standard, including Direct, GCG-T (greedy coordinate gradient attack), PAIR (iterative refinement attack), AutoDAN (automated jailbreak attack), HumanJB (human jailbreak attack), RFA (refusal to answer direction ablation attack), Prefill (prefix padding attack), and CodeAttack (code attack). The results are shown in Table 1.

[0066] Table 1. Comparison of Attack Success Rate (ASR) of Various Defense Methods under Different Attack Families On Llama-3-8B-Instruct, the ASR of the base model on RFA, Prefill, and CodeAttack were 79.4%, 67.8%, and 87.1%, respectively. After adopting the method of this invention, the ASR of the three ASRs decreased to 0.6%, 0.0%, and 0.0%, respectively, with relative reductions of approximately 99%, 100%, and 100%. The ASR of the other attack families (Direct, GCG-T, PAIR, AutoDAN, HumanJB) also decreased to 0.0%.

[0067] This invention achieved the lowest or tied lowest ASR among all 8 attack families.

[0068] The performance of this embodiment in terms of general capabilities and over-rejection rate is shown in Table 2. Among them, MMLU (Massive Multitask Language Understanding), MT-Bench (Multi-turn Dialogue Capability Evaluation Benchmark), and XSTest CR (XSTest Compliance Rate, which measures the degree of over-rejection of benign prompts by the model) are all commonly used evaluation metrics in this field.

[0069] Table 2 Comparison of various defense methods in terms of general capabilities and rejection rates (higher MMLU and MT-Bench are better, and higher XSTest CR indicates a lower rejection rate). The above results demonstrate that by explicitly bridging the existing harmful beliefs in the base model to the response initiation position and using belief gating to regulate the rejection direction during training, this embodiment can achieve near-complete robust defense against various jailbreak attacks, including rejection direction ablation, prefix padding, and code attacks, without significantly losing general capabilities or introducing over-rejection.

[0070] Example 2 This embodiment, based on Embodiment 1, further discloses the application of the method of the present invention on the Gemma-7B-IT (Google's open-source large language model, 7B parameter scale, IT represents the instruction fine-tuning version) large language model.

[0071] Gemma-7B-IT adopts<start_of_turn> / <end_of_turn> Dialogue template, t_inst is located in the user's turn.<end_of_turn> At the lexical level, t_post is located in the model round.<start_of_turn> After the lexical term "model".

[0072] The t_inst and t_post corresponding to the dialogue template are automatically located by special terms in the dialogue template, without the need to modify the method body.

[0073] For the Gemma-7B-IT model architecture, the preset layer set L is adjusted to L={10, 14, 18, 22}; the remaining hyperparameters are the same as in Example 1.

[0074] Experimental results: The ASR of the basic model on RFA, Direct, and CodeAttack were 92.8%, 28.7%, and 89.3%, respectively; after adopting the method of this invention, they decreased to 3.1%, 0.0%, and 0.0%, respectively, corresponding to relative reductions of 97%, 100%, and 100%.

[0075] MMLU improved from 52.0% to 52.5%, MT-Bench improved from 6.60 to 6.43, and XSTest improved from 89.2% to 88.0%.

[0076] The above results demonstrate that the method of the present invention also achieves significant defensive effects on the Gemma model, verifying the cross-model transferability of the method.

[0077] Example 3 Based on Embodiment 1, this embodiment further discloses the application of the method of the present invention on the Mistral-7B-Instruct-v0.2 (a large language model open sourced by Mistral AI, with 7B parameter scale and v0.2 being a fine-tuned instruction version) large language model.

[0078] Mistral-7B-Instruct-v0.2 uses the [INST] / [ / INST] dialog template, with t_inst located as the last user content term within [ / INST] and t_post located after [ / INST].

[0079] The t_inst and t_post corresponding to the dialogue template are automatically located by special terms in the dialogue template, without the need to modify the method body.

[0080] For the Mistral-7B-Instruct-v0.2 model architecture, the preset layer set L is L={12, 16, 20, 24}; the other hyperparameters are the same as in Example 1.

[0081] Experimental results: The ASR of the basic model on PAIR, AutoDAN, Prefill, and CodeAttack were 69.7%, 77.4%, 95.6%, and 86.8%, respectively; after using the method of this invention, they decreased to 3.8%, 3.8%, 0.0%, and 5.0%, respectively.

[0082] MMLU is consistent with the base model (58.9%), and the XSTest compliance rate is 92.0%, which is only 0.4pp lower than the base model's 92.4%, significantly better than existing methods (some baselines show a 7-12pp decrease in XSTest compliance rate on Mistral).

[0083] The above results demonstrate that the method of the present invention also achieves significant defense performance on the Mistral model and maintains a low rejection rate for benign prompts, thus verifying the versatility of the method.

[0084] Example 4 like Figure 4 As shown, based on the method of the present invention, this embodiment discloses a large language model security alignment system based on harmful belief-refusal behavior bridging. As an application system, the system divides the entire alignment process into eight functional modules through modular design, and each module is executed in series through data transmission connection.

[0085] The system includes the following modules: The data construction module is used to construct an aligned training dataset containing harmful prompts, benign prompts, and their harmful labels, and to identify inconsistent samples based on the response classification of the frozen basic large language model. In Example 1, this module collects 860 harmful prompts from a subset of AdvBench and SORRY-Bench, and generates 2348 benign prompts by GPT-4o, for a total scale of 3208. Inconsistent samples are identified based on the HarmBench evaluation rules and assigned a sample weight of ρ=2.0.

[0086] The anchor point estimation module is connected to the data construction module via a data transmission connection. It extracts the hidden states of the instruction end word t_inst and the response start word t_post on the frozen basic large language model, estimates the harmfulness direction v_inst, l, and the centroids of the four categories of harmful-benign and refuse-accept, and keeps them frozen during training, without participating in gradient updates. This module corresponds to step S2. The estimated anchor points remain fixed throughout the training process, improving training stability.

[0087] The scoring calculation module is connected to the anchor point estimation module via a data transmission connection. On the large language model with the low-rank adapter, the belief score B(x), bridging score M(x) and refusal score R(x) are calculated according to equations (1) to (3) respectively. This module corresponds to step S3. All three scores are scalars in the interval [-1, 1]. The arithmetic mean of the scores of each layer in the preset layer set is calculated.

[0088] The bridging and belief training module is connected to the scoring calculation module via a data transmission connection. Based on the bridging score M(x), the bridging loss L_bridge is calculated according to formula (4), and based on the belief score B(x), the belief loss L_bel is calculated according to formula (5). Both are in the form of weighted binary cross-entropy. This module corresponds to steps S4 and S5, and explicitly establishes a bridging channel from t_inst to t_post, so that the refusal decision is dominated by harmful belief signals.

[0089] The gated rejection module is connected to the bridging and belief training module via a data transmission connection. Based on the belief score B(x), a sample-level gate g(x, y) is constructed according to equation (6). The gate is multiplied sample by sample onto the binary cross-entropy corresponding to the rejection score R(x), and the gated rejection loss L_ref is obtained according to equation (7). This module corresponds to step S6. By stopping the gradient operator sg[·], the influence of backpropagation on the hidden state of the belief score is blocked, so that the training is only performed in full intensity on the rejection direction for samples whose beliefs are consistent with the labels, which significantly reduces the over-rejection rate.

[0090] The stability constraint module is connected to the gated rejection module via a data transmission connection. It calculates the benign KL loss L_kl on the benign prompt subset D_b according to equation (8), and calculates the language modeling loss L_lm based on the greedy decoding target of consistent samples. This module corresponds to step S7. The benign KL loss is only calculated on the benign subset to avoid retaining the undesirable acceptance behavior of the basic model on harmful prompts. The language modeling loss is reset to zero for inconsistent samples to avoid the reverse offsetting effect of other losses.

[0091] The joint optimization module is connected to the stability constraint module through a data transmission connection. It calculates the total loss L_total by weighted summation of bridging loss, belief loss, gated rejection loss, benign KL loss and language modeling loss according to equation (9), and only updates the low-rank adapter parameters. This module corresponds to step S8. In Example 1, λ_bel=λ_ref=λ_bridge=1.0, λ_kl=0.2, the optimizer is AdamW, the learning rate is 2e-4, and the training lasts for 3 epochs.

[0092] The adapter merging and deployment module is connected to the joint optimization module via a data transmission connection. After training is completed, the low-rank adapters are merged back into the frozen base large language model for inference deployment. This module corresponds to step S9. The merged deployment model is completely consistent with the base model in terms of parameter quantity, memory usage, and inference latency, and no modifications are required to the inference code.

[0093] The above eight modules are executed sequentially in the order of data construction → anchor point estimation → score calculation → bridging and belief training → gating rejection → stability constraints → joint optimization → adapter merging and deployment. Figure 4 As shown, the modules are connected in series through data transmission to form a complete large language model secure alignment system architecture, which can be deployed universally on various open source large language models such as Llama-3, Gemma, and Mistral.

[0094] Based on the above experimental results, the preferred ranges of key parameters in the method of the present invention are as follows: (1) The preferred range of loss weights λ_bel, λ_ref, and λ_bridge is 0.5-2.0, and the optimal value is 1.0; the preferred range of loss weights λ_kl is 0.1-0.5, and the optimal value is 0.2; (2) The preferred range of the gating scale coefficient α is 2-8, with the optimal value being 4.0: if α is too small, the gating will be too soft, causing conflicting samples to still participate in the training of the rejection direction, introducing over-rejection; if α is too large, the gating will be close to a hard threshold, causing the number of training samples to be reduced excessively, and the robustness will decrease. (3) The preferred range for the adjustment factor ρ of inconsistent samples is 1-5, with the optimal value being 2.0: if ρ=1, inconsistent samples will not be distinguished, and the correction will be insufficient; if ρ is too large, the training will overfit on a few inconsistent samples, which will impair the benign ability. (4) The preferred range of the preset layer set L is the middle layer of the large language model; the middle layer has the highest separability of harmful signals. The signals of the earlier layers have not yet been stably formed, and the later layers have been processed by tasks, which are not conducive to bridging training. (5) The preferred range of the LoRA adapter rank r is 8-32, with the optimal value being 16; if it is too large, the training cost and the risk of overfitting will increase, and if it is too small, the capacity will not be sufficient to support the five losses.

[0095] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several modifications and improvements can be made without departing from the inventive concept of the present invention, and these all fall within the protection scope of the present invention.

Claims

1. A large-scale model safety alignment method for bridging harmful beliefs and refusal-to-respond behavior, characterized in that, Performed by a computing device, which includes a processor and memory, the steps include: S1: Construct an alignment training dataset, including: obtaining harmful and benign prompts and their harmful labels; classifying the responses to the prompts based on the frozen basic large language model, and identifying inconsistent samples where the behavior of the basic model is inconsistent with the labels; the inconsistent samples include samples in which the basic model accepts harmful prompts and samples in which the basic model refuses to respond to benign prompts. S2: Estimating frozen anchor points includes: extracting hidden states at the end-of-user instruction and the beginning-of-response instruction positions on the preset layer set of the frozen basic large language model; calculating the mean of harmful and benign prompts in the hidden states at the end-of-user instruction positions using the processor; using the difference between the mean of harmful and benign prompts as the harmful direction; and using the mean of harmful prompts, the mean of benign prompts, the mean of rejection prompts, and the mean of acceptance prompts at the beginning-of-response instruction positions as four classification centroids; the harmful direction and the four classification centroids remain frozen during subsequent training; wherein the frozen basic large language model refers to a basic large language model whose model parameters remain unchanged during training; the word is a token; and the preset layer set is selected within the middle layer range of the frozen basic large language model. S3: Construct a low-rank adapter and define representation-level scores: Attach a low-rank adapter to the attention and feedforward projection of the frozen basic large language model; define belief scores, bridging scores, and rejection scores on a preset set of layers, and perform an arithmetic average of the scores of each layer. The belief score B(x), bridging score M(x), and refusal score R(x) are defined as follows: B(x) = (1 / |L|)·Σ_{l∈L}[cos(h_inst,l(x), μ + _inst,l) - cos(h_inst,l(x),μ - _inst,l)] M(x) = (1 / |L|)·Σ_{l∈L}cos(h_post, l(x), v_inst, l) R(x) = (1 / |L|)·Σ_{l∈L}[cos(h_post,l(x), μ + _post,l) - cos(h_post,l(x),m - _post,l)]; Where |L| represents the number of elements in the preset layer set L; Σ_{l∈L} represents the summation of all layer indices l in the preset layer set L, and (1 / |L|)·Σ_{l∈L}(·) is the arithmetic mean of the scores of each layer in L; h_inst,l(x) and h_post,l(x) mean: L is the preset layer set, l is the layer index; the hidden state vector of sample x at the l-th layer and t_inst position is denoted as h_inst,l(x), and the hidden state vector at the l-th layer and t_post position is denoted as h_post,l(x), both of which are the hidden dimension vectors of the layer; μ + _inst, l, μ - _inst, l, μ + _post, l, μ - The meanings of _post, l, and v_inst, l are: to calculate the average of the hidden states of harmful and benign prompts at position t_inst, respectively, to obtain μ. + _inst, l and μ - _inst, l, serves as the centroid for both harmful and benign categories; the harmful direction v_inst, l is defined as the difference between the mean of harmful and benign warnings. At the t_post position, the average of the rejected and accepted prompts in the base model is calculated to obtain μ. + _post, l and μ - _post, l, as the centroid for both rejection and acceptance; cos(a, b) represents the cosine similarity between vectors a and b; S4: Calculate the bridging loss, including: the processor passes the bridging score through the Sigmoid function and calculates a weighted binary cross-entropy with the harmfulness label; S5: Calculate the belief loss, including: the processor passes the belief score through the Sigmoid function and calculates a weighted binary cross-entropy with the harmful label; S6: Calculate the gated rejection loss, including: constructing a sample-level gate based on the belief score, wherein the gate is equal to applying a stopping gradient operator to the belief score, multiplying it by a sign corresponding to the label, then amplifying it by a scaling factor and taking the sigmoid; multiplying the gate sample by sample onto the binary cross-entropy corresponding to the rejection score to obtain the gated rejection loss; S7: Calculate the benign KL loss and language modeling loss, including: on a subset consisting only of benign prompts, calculate the KL divergence as the benign KL loss based on the probability distribution of the next word in the base language model and the model with the low-rank adapter after freezing the initial word position of the response; for samples whose behavior is consistent with the label in the base model, calculate the next word cross-entropy as the language modeling loss based on the target response obtained by greedy decoding of the frozen base language model; the greedy decoding refers to selecting the word with the highest probability as the output in each generation step. S8: Joint optimization, including: linearly weighting the bridging loss, belief loss, gated rejection loss, benign KL loss and language modeling loss according to preset non-negative weights to obtain the total loss, and performing joint optimization only on the low-rank adapter parameters; the preset non-negative weights include bridging loss weight, belief loss weight, gated rejection loss weight, benign KL loss weight and language modeling loss weight; S9: Adapter merging deployment, including: after training, merging the low-rank adapter back into the frozen base large language model and deploying it with the same number of parameters and inference latency as the frozen base large language model.

2. The large-scale model secure alignment method for bridging harmful beliefs and refusal to respond, as described in claim 1, is characterized in that... In step S1, the sample weight of the inconsistent sample is 2 to 5 times that of the non-inconsistent sample.

3. The large-scale model secure alignment method for bridging harmful beliefs and refusal to respond, as described in claim 1, is characterized in that... In step S2, the position of the last word of the instruction is the position of the last word of the user instruction in the dialogue template, and the position of the first word of the response is the position of the boundary word before the first word of the assistant response in the dialogue template; for different dialogue templates, the position of the last word of the instruction and the position of the first word of the response are automatically located according to the special words corresponding to the template in the large language model.

4. The large-scale model secure alignment method for bridging harmful beliefs and refusal to respond, as described in claim 1, is characterized in that... In step S7, the benign KL loss is calculated only on the benign cue subset, and no KL constraint is imposed on the harmful cue subset, so as to avoid retaining undesirable acceptance behaviors of the frozen basic large language model on harmful cue.

5. The large-scale model safety alignment method for bridging harmful beliefs and refusal to respond according to claim 1, characterized in that, In step S7, the language modeling loss uses the greedy decoding result of the frozen basic large language model on the training prompt as the token-level supervision target. For samples whose model behavior is inconsistent with the label, the sample weight of the language modeling loss is reset to zero, so as to avoid the language modeling loss from inversely offsetting the corrective effect of bridging loss, belief loss and gating rejection loss on the inconsistent samples.

6. The large-scale model secure alignment method for bridging harmful beliefs and refusal to respond according to claim 1, characterized in that, In step S2, the preset layer set is selected from the intermediate layers of the large language model. For the Llama-3 and Mistral series, layers 12, 16, 20, and 24 are used, and for the Gemma series, layers 10, 14, 18, and 22 are used. For other series of models, four intermediate layers are selected proportionally according to the number of layers, so that the relative position of the selected layer in the total number of layers in the model corresponds to the relative position of the selected layer in the reference model.

7. The large-scale model secure alignment method for bridging harmful beliefs and refusal to respond according to claim 1, characterized in that, In step S8, the preset non-negative weights include: bridging loss weight of 0.5 to 2.0, belief loss weight of 0.5 to 2.0, gating rejection loss weight of 0.5 to 2.0, benign KL loss weight of 0.1 to 0.5, and language modeling loss weight of 1.

0.

8. A safe alignment system for a large language model based on harmful beliefs and refusal-to-respondence behavior bridging, characterized in that, Performing the alignment method of claim 1 includes: The data construction module is used to build an aligned training dataset containing harmful prompts, benign prompts and their harmful labels, and to identify inconsistent samples based on the response classification of the frozen base large language model. Anchor point estimation module, connected to the data construction module via data transmission connection, is used to receive data output by the data construction module, extract the hidden states of instruction end words and response start words on the frozen basic large language model, estimate the harmfulness direction and four classification centroids, and keep them frozen during training; The scoring calculation module is connected to the anchor point estimation module via a data transmission connection. It is used to receive the data output by the anchor point estimation module and calculate the belief score, bridging score and refusal score on the large language model with the low-rank adapter attached. The bridging and belief training module is connected to the scoring calculation module via a data transmission connection. It is used to receive the data output by the scoring calculation module and calculate the bridging loss and belief loss in the form of weighted binary cross-entropy based on the bridging score and belief score, respectively. The gated rejection module is connected to the bridging and belief training module via a data transmission connection. It is used to receive the data output by the bridging and belief training module, construct sample-level gating based on belief scores, and multiply the gating sample by sample by the binary cross-entropy corresponding to the rejection score to obtain the gated rejection loss. The stability constraint module is connected to the gated rejection module via a data transmission connection. It is used to receive the data output by the gated rejection module, calculate the benign KL loss on the benign prompt subset, and calculate the language modeling loss based on the greedy decoding target of consistent samples. The joint optimization module is connected to the stability constraint module via a data transmission connection. It receives the data output by the stability constraint module, performs a weighted summation of bridging loss, belief loss, gated rejection loss, benign KL loss, and language modeling loss, and updates only the low-rank adapter parameters. The adapter merging and deployment module is connected to the joint optimization module via a data transmission connection. It is used to receive the data output by the joint optimization module and merge the low-rank adapter back into the frozen base large language model for inference deployment after training is completed. The modules are executed sequentially in the order of data construction, anchor point estimation, score calculation, bridging and belief training, gating rejection, stability constraints, joint optimization, and adapter merging deployment. The above eight functional modules are connected in series through data transmission to form a complete large language model secure alignment system architecture.

Images