Large language model security alignment method based on autoregressive reward guidance and representation editing

Abstract

The invention discloses a large language model security alignment method based on autoregressive reward guidance and representation editing. The method comprises the following steps: firstly, bidirectionally identifying a key intervention target spot through gradient attribution, wherein the key intervention target spot comprises a toxic driver word in a toxic sample and a safe anchor word in a non-toxic sample; then, extracting a pure global security guide vector through principal component analysis by masking a target vocabulary and calculating a hidden state difference; then, training an autoregressive reward model capable of providing dense lexical element level reward signals; and finally, during reasoning generation, calculating and generating an award in real time, and dynamically adjusting the hidden state of the model along the security guide vector to maximize the security award so as to generate a response meeting the security requirement. According to the method, a basic model does not need to be retrained, efficient, flexible and accurate safety intervention is realized in a reasoning stage, semantic distortion is effectively avoided, and the universality of the model is reserved to the maximum extent.

Description

Technical Field

[0001] This invention relates to the field of large language model secure alignment technology, specifically to a model secure alignment method based on autoregressive reward guidance and dynamic representation editing. Background Technology

[0002] With the widespread application of large language models, the security of their generated content has become a key challenge. Existing methods for improving model security can be mainly divided into two categories: alignment during training and intervention during inference, but both have significant drawbacks.

[0003] First, alignment methods during training, such as reinforcement learning based on human feedback and its improved algorithms, such as direct preference optimization, align the model parameters with human preferences by fine-tuning them. However, these methods have fundamental limitations: first, they rely on large-scale, high-quality preference data and expensive full-parameter training, making them costly; second, once training is complete, the model behavior is fixed and cannot flexibly adapt to new, undefined security requirements during the inference phase, lacking the ability to dynamically adjust.

[0004] Secondly, intervention-time methods attempt to address the aforementioned flexibility issues, but introduce new challenges in their implementation. This type of method can be further subdivided into:

[0005] 1. Guided Decoding: This approach avoids harmful content by adjusting the probability distribution of lexical units during the decoding process. However, existing methods heavily rely on trajectory-level reward models, which score the generated complete or partial responses. Since the reward model itself is trained on complete responses, using it to evaluate incomplete partial responses leads to inaccurate reward signals. Generating complete subsequent trajectories for each candidate lexical unit for evaluation to avoid inaccuracy incurs prohibitively high inference costs.

[0006] 2. Weight Editing: This aims to remove "toxic" components by directly modifying model weights. However, in large-scale language models, harmful knowledge is closely coupled with general knowledge. Aggressive weight editing often impairs the model's general language capabilities, making it difficult to balance detoxification effectiveness with model performance.

[0007] 3. Representation Editing: Intervening at the hidden representation level of the model is considered a more promising direction. However, existing methods have insufficient exploration of the dynamic transition laws of "toxicity" in the representation space, resulting in coarse and imprecise intervention signals and limited effectiveness.

[0008] In summary, existing technologies cannot achieve accurate, flexible, efficient, and secure alignment of large language models without retraining the model, sacrificing general capabilities, and maintaining efficient inference. Therefore, there is an urgent need in this field for a new technical solution to overcome these shortcomings. Summary of the Invention

[0009] To address the problems of high cost and poor flexibility in existing alignment methods during training, and inaccurate reward signals, high inference overhead, and imprecise representation editing direction in inference-time intervention methods, this invention provides a secure alignment method for large language models based on autoregressive reward guidance and representation editing. This invention aims to achieve efficient, flexible, and secure text generation without compromising the model's general capabilities by constructing a clean semantically safe direction, training a dense lexical-level reward model, and performing dynamic and precise representation editing during inference, without retraining the basic large language model.

[0010] To achieve the above objectives, the technical solution of the present invention comprises the following four core steps in sequence:

[0011] 1) Dataset Construction and Safe Guidance Vector Extraction. A pairing dataset was constructed based on Wikitext-2 hints. By identifying key terms (toxicity-driven terms in toxic samples and safety-anchoring terms in non-toxic samples) through gradient attribution, a global safety guidance vector s is extracted from the differences in their hidden states. This vector defines the core direction leading to safe content in the model representation space.

[0012] 2) Autoregressive reward model training. Train a word-level autoregressive reward model θ. r It decomposes the total reward of the sequence into the sum of the rewards of each word. By optimizing a composite objective function that combines preference loss and numerical stability constraints, the model can assign higher rewards to safe words, providing fine-grained real-time guidance signals for the generation process.

[0013] 3) Dynamic reasoning based on reward and security vectors. Reward guidance and representation editing are performed when the model generates each lexical unit. Reward guidance involves calculating the reward for the current generation path and comparing it with the target security reward R. target Comparison. This indicates that, when a reward gap exists, the hidden state h of the model is adjusted along the direction of the safety guidance vector s. t Dynamic adjustments are made, which can be formalized as follows: Among them, R(h) t ) is the reward score for the current hidden state.

[0014] 4) Generation and output of security responses. After the above dynamic guidance and state editing, the model finally outputs text content that is relevant to the user prompts and conforms to security specifications, completing end-to-end security alignment from the generation source to the final output.

[0015] Compared with the prior art, the beneficial effects of the present invention include:

[0016] 1. Precision of intervention: By accurately locating toxic words through gradient attribution and combining them with pure and safe guiding vectors, intervention measures are targeted and effectively avoid semantic distortion.

[0017] 2. Efficiency and flexibility: The entire solution is completed during the inference phase, eliminating the need for expensive retraining of the base model, and the degree of safe alignment can be flexibly controlled by adjusting the guidance strength α.

[0018] 3. High signal quality: The autoregressive reward model provides dense word-level reward signals, overcoming the problems of sparse and delayed signals in the trajectory-level reward model, making the guidance of the generation process more timely and stable.

[0019] 4. High versatility: This method does not modify the model weights, thus preserving the general capabilities of the basic model to the greatest extent. While achieving effective detoxification, it ensures the model's performance on other tasks. Attached Figure Description

[0020] Figure 1 The above is the overall flowchart of this invention. It includes four steps: gradient attribution-based toxic word identification, representation space safety guidance vector, training of the autoregressive reward model, and dynamic inference of the integrated autoregressive reward model.

[0021] Figure 2 This is a schematic diagram of the representation space security guidance vector generation of the present invention.

[0022] Figure 3 This is the dynamic inference graph of the integrated autoregressive reward model of the present invention. Detailed Implementation

[0023] The following detailed description, in conjunction with the accompanying drawings, illustrates the technical solutions of the present invention to make the objectives, technical solutions, and advantages of the present invention clearer. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it.

[0024] This invention provides a secure alignment method for large language models based on autoregressive reward guidance and representation editing. The core process is as follows: Figure 1 As shown (Note: This assumes...) Figure 1 This is a flowchart of the overall method (the actual writing should refer to the attached diagram), which specifically includes the following steps S1 to S4.

[0025] Step S1: Toxicity Lexical Identification Based on Gradient Attribution (Precisely Locating Intervention Targets). This step aims to precisely locate lexical units in the input text that have a key impact on the model's safety decisions, including negatively impacting words that drive toxicity and positively impacting words that inhibit toxicity, thus serving as intervention targets for subsequent representation editing. The specific implementation process is as follows:

[0026] Step S11: Obtain the toxicity score. This method is based on a text dataset constructed from samples from Wikitext-2. Each cue corresponds to a pair of data samples x: toxic samples are generated using basic GPT-2 greedy decoding, while non-toxic samples are generated using PPLM (Plug-and-Play Language Models) technology, guided by a predefined non-toxic attribute classifier. For each pair of samples x, the toxicity probability P is calculated using the Detoxify model. toxicity (x)∈[0,1].

[0027] Step S12: Calculate the gradient attribution score. Based on P toxicity (x), calculate the lexical embedding e j Attribution score:

[0028]

[0029] Where j represents the nth word in the sequence; i represents the nth dimension in the word vector; and H is the dimension of the word vector. Indicates the embedding of the lexical element e j Find the partial derivative. It is the logarithmic probability form of the toxicity probability P, and ⊙ represents the element-wise multiplication of the vectors;

[0030] For toxic sample x toxic The K morphemes with the highest attribution scores are selected as the toxicity-driven word set I. + After the mask For non-toxic sample x non-toxic The tokens with the lowest attribution scores are selected as the safe anchor word set I. - After the mask

[0031] Step S2: Constructing a Safe Guiding Vector in the Representation Space (Extracting a Pure Safe Direction). This step aims to extract a pure semantic vector from the model's hidden state space that can effectively guide text generation towards a safe direction. Specifically, as shown... Figure 2 As shown, the method for constructing a spatially secure guidance vector includes the following steps:

[0032] Step S21: Construct a single-sample semantic transformation vector.

[0033] In this step, for each sample x in dataset D and its identified set of key toxic terms I... toxic Furthermore, it is classified into keywords I with significant positive impact. + And keywords I with significant negative impact -To ensure the vocabulary set contains lexical units with substantial semantic contributions, we pre-removed stop words and other functional words from the candidate words before constructing the vocabulary set. Then, the hidden state difference vector Δ(x) generated in the last layer of the model is calculated as follows:

[0034] 1. Change the original input x non-toxic I in + The tokens are replaced with the special padding character [PAD] to obtain the modified input x. toxic\+ And obtain the l-th hidden state corresponding to its last word element.

[0035] 2. Similarly, change the original input x toxic I in - Replace the morpheme with [PAD] to get x toxic\- and obtain h last (x toxic\- ).

[0036] 3. Calculate the difference vector:

[0037]

[0038] Among them, h last (·) is the semantic vector of the last token in the last layer of the target model;

[0039] This vector captures the net change direction of the model's hidden state in the semantic space after removing keywords with positive and negative effects, and can be regarded as a safe semantic transformation direction for a single sample.

[0040] Step S22: Directional aggregation to extract the global safety guidance vector. Collect the set of difference vectors {Δ(x)|x∈D} calculated at the l-th layer for all samples in dataset D. Perform principal component analysis on this set and extract its first principal component:

[0041] s (1) =PCA ) ({Δ(x)∣x∈D})

[0042] Among them, s (1) This is the final global security guidance vector, PCA. ) (·) indicates that principal component analysis is performed on the input vector set and its first principal component is extracted.

[0043] The obtained principal component vector s (1) This refers to a clean global security guidance vector extracted and aggregated from a large amount of data. It represents the most significant common direction of the dataset toward security semantic shift.

[0044] Step S3: Autoregressive Reward Model Training. This step aims to train an autoregressive reward model that can provide fine-grained, word-level reward signals, offering accurate and timely feedback for dynamic guidance during inference.

[0045] Step S31: Model Structure and Reward Definition. Autoregressive Reward Model θ r A two-layer, multilayer perceptron architecture is employed. Its input is the context-aware hidden state of the model when generating the t-th lexical unit. The model output is the immediate reward value for that word. The total reward for the entire sequence (p,y) is defined as the sum of the rewards for all words:

[0046]

[0047] Where p is the input text of length M; y is the partial response of length T; and t represents the time step in the autoregressive generation process. is the context-aware hidden state when generating the t-th lexical; r(p,y) is the sum of the rewards for the generated lexical.

[0048] Step S32: Loss Function Design and Optimization. A dataset D containing pairs of non-toxic / toxic texts is used. h Training is performed. For non-toxic data (hidden state sequence h)... + ) and toxic data (hidden state sequence is h) - Define preference loss:

[0049]

[0050] Where σ is the Sigmoid function, β r It is the temperature coefficient.

[0051] To stabilize the reward value, an additional regularization term is introduced. Let r - =∑ t θ r (h - [t]), r + =∑ t θ r (h + [t]), the total loss function is:

[0052]

[0053] Where, τ neg and τ pos λ is the reward threshold. abs and λ ext These are the corresponding weighting coefficients. Through optimization... To train the reward model parameters θ r .

[0054] Step S4: Dynamic Inference with Integrated Autoregressive Reward Model (Real-time Guided Generation). In this step, during text generation through model inference, the hidden states of the model are edited dynamically and in real-time using the safety guidance vector and reward model obtained in the previous steps. Specifically, as follows... Figure 3 As shown.

[0055] Step S41: Forward Propagation and Reward Calculation. Given an input cue word, the model is started to perform autoregressive generation. When generating the t-th lexical unit, the model performs forward propagation to obtain the hidden state at the current time step. (l is the last hidden layer). Input this hidden state into the trained autoregressive reward model θ. r To obtain the reward estimate for the current generation step.

[0056] Step S42: Optimize the representation of reward guidance. Set a target security reward value R. target Compare the current reward R(h) t ) and target reward R target .

[0057] If there is a reward gap (i.e., R(h) t ) < R target Then, along the safety guidance vector s extracted in step S2 (1) The direction of the current hidden state Adjustments will be made:

[0058]

[0059] Where α is the guiding strength coefficient, this optimization process can be carried out iteratively through gradient descent to maximize the reward; This is the final hidden state after adjustments; R target Defined as the average token-level reward for all preferred responses in the dataset; N + x represents all non-toxic samples in the dataset non-toxic Total quantity; h last (x non-toxic ) (1) Indicates x non-toxic After inputting the model, the hidden state vector corresponding to the last word in the last layer.

[0060] Step S43: Generate a security response. Use the adjusted hidden state. Replace the original Continue with the subsequent lexical prediction and generation process. Repeat steps S41 to S43 until a complete response is generated.

[0061] Through the synergistic effect of steps S1 to S4 above, this invention can achieve efficient, flexible, and accurate secure text generation without retraining the basic large language model, while preserving the original general capabilities of the model to the greatest extent.

[0062] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A secure alignment method for large language models based on autoregressive reward guidance and representation editing, characterized in that, include: A. Identify positive and negative driving words in the input samples based on gradient attribution, and construct a safe guiding vector in the representation space; B: Train a lexical-level autoregressive reward model. During the large language model inference generation process, by integrating the lexical-level autoregressive reward model with a safety guidance vector, the hidden state of the model generation process is guided in real time from the trajectory pointing to harmful content to the safe content area, thereby outputting response text that conforms to safety specifications.

2. The secure alignment method for large language models based on autoregressive reward guidance and representation editing according to claim 1, characterized in that, Step A includes: A1. Using sentences from the Wikitext-2 dataset as prompts, generate a pair of data samples x for each prompt, including toxic samples x. toxic and non-toxic sample x non-toxic Toxic samples are generated by the base GPT-2 model through greedy decoding, while non-toxic samples are generated by PPLM (Plug-and-Play Language Models) technology, guided by a predefined non-toxic attribute classifier; thus forming dataset D. A2. For each pair of data samples x, first use an independent toxicity scoring model, Detoxify, to evaluate them and obtain a comprehensive toxicity probability score P. toxicity (x)∈[0,1]; then calculate the embedding e for each lexical unit. j Attribution score a of toxicity score j The calculation formula is: Where j represents the nth word in the sequence; i represents the nth dimension in the word vector; and H is the dimension of the word vector. Indicates the embedding of the lexical element e j Find the partial derivative. It is the logarithmic probability form of the toxicity probability P, and ⊙ represents the element-wise multiplication of the vectors; A3. For toxic sample x toxic Select the attribution score a. j The top K words constitute the key positive driving word set I of this sample. + By replacing these tokens with padding characters, a masked sample is constructed. For non-toxic sample x non-toxic Select the attribution score a. j The lowest K words constitute the key negative modifier set I of this sample. + By replacing these tokens with padding characters, a masked sample is constructed. A4, x toxic x non-toxic and its mask variants Given the target large language model to be securely aligned, extract the hidden state of the last token in the last layer of the model and calculate the difference vector Δ(x): Among them, h last (·) is the semantic vector of the last token in the last layer of the target model; the difference vector Δ(x) in the representation space of the target model quantifies the core evolution direction from toxic representation to non-toxic representation; A5. Perform principal component analysis (PCA) on the difference vector Δ(x) calculated for all samples x in the dataset and placed in the target model representation space, and extract its first principal component as the global safety guiding vector: s (l) =PCA1({Δ(x)∣x∈D}) Among them, s (l) The final global safety guidance vector is obtained. PCA1(·) performs principal component analysis on the input vector set and extracts its first principal component.

3. The secure alignment method for large language models based on autoregressive reward guidance and representation editing according to claim 1, characterized in that, Step B includes: B1. Training the autoregressive reward model θ r This is a word-level model that decomposes the total reward of a sequence into the sum of the rewards of each word. For a sequence (p, y), the total reward is: Where p is the input text with a length of M; y is a partial response with a total response length of T; t represents the time step in the autoregressive generation process; and h... p,y≤t [M+t] is the context-aware hidden state when generating the t-th lexical unit; r(p,y) is the sum of the rewards for the generated lexical units; The total loss function of the autoregressive reward model is: in For preference loss; σ(·) is the sigmoid function, which maps the difference in cumulative rewards to the model's decision h. + Better than h - The probability of preference; β r It is a scaling hyperparameter used to control the sensitivity of reward differences to preference probabilities; r _ and r + The total reward for non-toxic samples and toxic samples, respectively; τ neg and τ pos λ is the reward threshold; abs and λ ext These are the weighting coefficients; B2. When generating each word, obtain the current hidden state of the model. The current reward R(h) is calculated using the autoregressive reward model. t ); B3. The current reward R(h) t ) and the preset target security reward R target Compare; B4. When a reward gap exists, along the safety guidance vector s (l) The direction relative to the current hidden state Adjustments are made, and the adjustment formula is as follows: Where α is the guiding strength coefficient, this optimization process can be carried out iteratively through gradient descent to maximize the reward; This is the final hidden state after adjustments; R target Defined as the average token-level reward for all preferred responses in the dataset; N + x represents all non-toxic samples in the dataset non-toxic Total quantity; h last (x non-toxic ) (l) Indicates x non-toxic After inputting the model, the hidden state vector corresponding to the last word in the last layer; B5. Use the adjusted hidden state. Perform subsequent lexical predictions until a response text that conforms to security specifications is generated.

Images