An interpretable reward model construction method based on a sparse self-encoder

Abstract

The application provides an interpretable reward model construction method based on a sparse self-encoder, and the method comprises the following steps: constructing and pre-training a sequence-level sparse self-encoder, and constructing an interpretable reward model, wherein the encoder part of the pre-trained sparse self-encoder is integrated after the first layer of a basic language model, and all original layers after the first layer of the basic language model are discarded; the reward model is trained based on preference data, wherein the encoder parameters of the sparse self-encoder are frozen, the parameters of the front layers of the basic language model and the weights of a linear value head are trained, the weights of the linear value head are optimized by using a pair of preference data through a preference learning loss function, and the model learns the contribution weight of each interpretable feature to human preference. The method can be simply and efficiently applied to various real scenes, and is particularly suitable for reinforcement learning training of a strategy model under dynamic preference and large-scale data filtering.

Description

Technical Field

[0001] This application relates to the field of large language model technology, specifically a method for constructing an interpretable reward model based on a sparse autoencoder. Background Technology

[0002] Large Language Models (LLMs) have been widely applied across various fields. To align the behavior of LLMs with human values, Human Feedback Reinforcement Learning (RLHF) has become a mainstream paradigm. In RLHF, the reward model (RM) acts as an agent of human preferences, guiding policy optimization by assigning numerical rewards to the model output. Therefore, the accuracy, reliability, and interpretability of the reward model are crucial for the alignment effect of downstream models.

[0003] Existing reward model techniques mainly suffer from the following shortcomings: Reward scores are difficult to interpret: Traditional scalar reward models (RMs) only output a scalar score, which cannot explain the source of the score or the reasons why the model prefers a certain response. This opacity makes it difficult for researchers to distinguish whether the model truly aligns with human values ​​or simply takes advantage of spurious correlations in the training data.

[0004] Inflexible preference adjustment: Once trained, the preferences of a traditional RM are fixed. If user preferences shift (e.g., to meet stricter security standards), it is usually necessary to collect new data and retrain the model, making dynamic adjustment impossible.

[0005] In summary, existing improvement schemes are costly and still have limitations: although multidimensional reward models (RMs) attempt to improve interpretability by predicting scores for multiple dimensions (such as usefulness and security), this introduces two new problems: first, it requires expensive and complex fine-grained multidimensional labeled data; second, even with dimensional scores, feature-level attribution remains opaque (i.e., we know that "security" is low, but we don't know which specific word or concept caused the low security).

[0006] Against this backdrop, designing a reward model that does not require additional multidimensional annotation costs, provides feature-level interpretability, and supports dynamic intervention and control is a technical problem that urgently needs to be solved in the current RLHF field. Summary of the Invention

[0007] The problem addressed by this invention is how to design a reward model that does not require additional multidimensional annotation costs, provides feature-level interpretability, and supports dynamic intervention and control.

[0008] To address the aforementioned issues, this invention provides a method for constructing an interpretable reward model based on a sparse autoencoder, a model product, a training system, and a storage medium.

[0009] In a first aspect, the present invention provides a method for constructing an interpretable reward model based on a sparse autoencoder, comprising the following steps: Construct and pre-train a sequence-level sparse autoencoder, wherein the input sequence is fed into the pre-trained language model. The hidden activation vector of the last word is extracted as input, and the hidden activation vector is mapped to a high-dimensional sparse feature space through a sparse autoencoder to obtain a sparse feature vector. Unsupervised pre-training is then performed with the goal of minimizing the reconstruction error. Construct an interpretable reward model, in which the encoder part of a pre-trained sparse autoencoder is integrated into the base language model. After the first layer, discard the base language model. All original layers after the layer, and construct a linear value head on the sparse feature vector to weight and aggregate the sparse features into a scalar reward; A reward model is trained based on preference data, wherein the encoder parameters of the sparse autoencoder are frozen, and the base language model is trained using the pre-processor. The layer parameters and the weights of the linear value heads are optimized using a preference learning loss function with paired preference data, enabling the model to learn the weight of each interpretable feature's contribution to human preferences.

[0010] Optionally, the sparse autoencoder is a TopK sparse autoencoder, and its encoding process is represented as follows: ,in The parameter representing the pre-bias term in the sparse autoencoder. The hidden state vector; The decoding and reconstruction process is represented as follows: ; in and These are the encoding matrix and the decoding matrix, respectively. For bias, for The sparse feature vector is used, and the TopK operation is used to force the retention of the K features with the largest activation values, while setting the rest to zero; the loss function for the reconstruction error is: , Let represent the hidden state vector after reconstruction by the sparse autoencoder.

[0011] Optionally, the linear value head calculates the scalar reward by weighting and aggregating sparse features using the following formula: ,in For the first The activation strength of a sparse feature The learnable weights are the corresponding features.

[0012] Optionally, the preference learning loss function is a pairwise ranking loss based on the Bradley-Terry model: ; in, This represents all parameters of the reward model. The reward model is based on the question. With answer The reward points given To win the reply, This is a reply indicating failure. This is the sigmoid function.

[0013] Optionally, it also includes: During the inference phase, for any input, the scalar reward output by the reward model is decomposed into a product of a set of activated sparse features and their corresponding weights to achieve feature-level attribution. Dynamic preference intervention can be achieved by directly modifying the weights of specific features without retraining the model.

[0014] Optionally, the dynamic preference intervention specifically involves: identifying a set of features that are positively correlated with the target preference, and multiplying the weights corresponding to the feature set by an amplification factor during inference, so as to guide the model to give higher rewards to outputs that conform to the target preference.

[0015] Secondly, embodiments of the present invention provide an interpretable reward model product based on a sparse autoencoder, comprising: The basic language model module is used to extract semantic representations of the input sequence, its preceding... Layer parameters can be frozen or fine-tuned; The sequence-level sparse autoencoder module is integrated into the basic language model module. After the layer, it is used to transfer the first The hidden activation vector of the last word in the layer is mapped to a high-dimensional sparse feature vector, and its encoder parameters are frozen during the training phase of the reward model. The linear value head module, connected after the sparse autoencoder module, is used to weight and aggregate the sparse feature vectors into a scalar reward, the weights of which are learnable parameters obtained by training with preference data. Among them, the basic language model module number All original layers after the first layer are discarded.

[0016] Optionally, the sparse autoencoder module is a TopK sparse autoencoder, and its encoding process is as follows: The decoding and reconstruction process is as follows: ;in and These are the encoding matrix and the decoding matrix, respectively. For bias, Given a sparse feature vector, the TopK operation is used to force the retention of the K features with the largest activation values.

[0017] Thirdly, embodiments of the present invention provide a training system for an interpretable reward model based on a sparse autoencoder, comprising: The pre-training unit is used to perform unsupervised pre-training of sequence-level sparse autoencoders using a general corpus, with the goal of minimizing reconstruction error; The model building unit is used to integrate the pre-trained sparse autoencoder into the base language model, discard subsequent layers, and add linear value headers. Preference training units are used to train the base language model by freezing the encoder parameters of the sparse autoencoder using pairwise preference data. The layer parameters and the weights of the linear head are optimized using a preference-learning loss function.

[0018] Fourthly, embodiments of the present invention provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the interpretable reward model construction method based on a sparse autoencoder as described in the first aspect.

[0019] The beneficial effects of the interpretable reward model construction method based on sparse autoencoder of the present invention are as follows: This method first pre-trains the sequence-level hidden states of the intermediate layers of a basic language model using a general corpus. A sparse autoencoder maps the originally opaque latent space into a sparse, unambiguous, and interpretable feature space. Then, the pre-trained SAE encoder is integrated into the reward model architecture, replacing subsequent layers of the basic model, and the encoder parameters are frozen. Next, linear value heads are used to weight and aggregate the extracted sparse features. Without requiring expensive multi-dimensional supervised annotations, feature weights can be trained using only pairwise preference data through a standard loss function. Finally, the model not only outputs a scalar reward but also directly attributes the reward score to specific semantic features. This method can be easily and efficiently applied to various real-world scenarios, and is particularly suitable for reinforcement learning training of policy models under dynamic preferences and large-scale data filtering.

[0020] This invention eliminates the need to retrain the reward model; by simply adjusting the weights of specific features in the linear value head, the reward signal can be changed in real time, guiding the policy model towards new preferences. In large-scale data filtering, this invention utilizes interpretable feature scores to accurately identify and filter high-quality data that meets specific semantic requirements from massive datasets. This data can then be used for fine-tuning or pre-training of downstream models. It is not only an effective method to improve the model's expressive power, discriminative ability, and accuracy, but also possesses extremely high flexibility. Attached Figure Description

[0021] Figure 1 This is a flowchart of the method for constructing an interpretable reward model based on a sparse autoencoder in an embodiment of the present invention; Figure 2 This is a schematic diagram of the method for constructing an interpretable reward model based on a sparse autoencoder in an embodiment of the present invention. Figure 3 This is a performance comparison chart of different reward models in the RewardBench 2 benchmark test in the embodiments of the present invention; Figure 4 This is a block diagram of the training system for an interpretable reward model based on a sparse autoencoder, as described in an embodiment of the present invention. Detailed Implementation

[0022] To better understand the purpose, technical solution, and advantages of this application, the application is described and explained below in conjunction with the accompanying drawings and embodiments.

[0023] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.

[0024] like Figure 1 and Figure 3 As shown in the embodiment of the present invention, a method for constructing an interpretable reward model based on a sparse autoencoder includes the following steps: S1. Construct and pre-train a sequence-level sparse autoencoder, wherein the input sequence is fed into the pre-trained language model. The hidden activation vector of the last word is extracted as input, and the hidden activation vector is mapped to a high-dimensional sparse feature space through a sparse autoencoder to obtain a sparse feature vector. Unsupervised pre-training is then performed with the goal of minimizing the reconstruction error. In this embodiment, the sparse autoencoder is a TopK sparse autoencoder.

[0025] Specifically, in order to extract high-level semantic features related to decision-making, rather than superficial word-level patterns, this invention adopts a sequence-level pre-training strategy.

[0026] Step 1.1: Extract sequence-level hidden states: Given an input sequence The input is fed into a pre-trained language model, and its intermediate layer is extracted. The hidden layer activation. Unlike traditional word-level pre-training, this invention only extracts the activation vector of the last "to" word in the sequence. It serves as input to SAE because this location gathers the abstract semantic information of the entire sentence.

[0027] ; In the formula, Indicates LLM before All parameters of the layer decoder layer, Indicates the last word.

[0028] Step 1.2, Constructing a TopK Sparse Autoencoder Model (TopK SAE): Using a TopK sparse autoencoder... Encoding and reconstruction are then performed. A sparse autoencoder (SAE) maps low-dimensional hidden states to a high-dimensional sparse feature space. Specifically: Encoding process (obtaining sparse features): ; The parameter representing the pre-bias term in the sparse autoencoder; Refactoring process: ; in, and These are the encoding and decoding matrices, respectively. For bias, Let M be the latent feature vector, and M be the feature dimension (much larger than the input dimension d). The TopK operation forces the retention of the K features with the largest activation values, and sets the rest to zero.

[0029] Step 1.3, Training the SAE: Use a general corpus to perform unsupervised training on the SAE to minimize the reconstruction error. ; In the formula, The hidden state vector. This represents the hidden state vector after SAE reconstruction.

[0030] After training, each feature of SAE corresponds to a single, interpretable concept.

[0031] S2. Construct an interpretable reward model, wherein the encoder part of the pre-trained sparse autoencoder is integrated into the base language model. After the first layer, discard the base language model. All original layers after the first layer, and construct a linear value head on top of the sparse feature vector, weighted and aggregated the sparse features into a scalar reward; The formula for calculating the scalar reward by weighting and aggregating sparse features using a linear value head is as follows: ,in For the first The activation strength of a sparse feature The learnable weights are the corresponding features.

[0032] Specifically, construct the SARM reward model architecture (an interpretable reward model). Step 2.1, Model Integration: Insert the pre-trained SAE encoder into the l-th layer of the base LLM.

[0033] Step 2.2, Hierarchical truncation and discarding of basic LLM All subsequent original layers are directly based on the sparse feature vectors output by SAE. Perform the calculation.

[0034] Step 2.3: Constructing the linear value head: In the SAE feature vector A learnable linear layer is added on top to aggregate features into a scalar reward.

[0035] ; in, It is the first The activation intensity of each feature These are the learnable weights corresponding to the feature. This linear combination guarantees the complete additivity and interpretability of the reward score.

[0036] S3. Train a reward model based on preference data, where the encoder parameters of the sparse autoencoder are frozen, and the base language model is trained. The layer parameters and the weights of the linear value heads are optimized using a preference learning loss function with paired preference data, enabling the model to learn the weight of each interpretable feature's contribution to human preferences.

[0037] For example, the preference data format is as follows: ,in This represents the user's input (Query). , The answer represents the model's survival, where The answer was selected as the better one by human experts. The answer was chosen as the worse by human experts.

[0038] Training a reward model based on preference data includes: Step 3.1, Freeze Parameters: During training, freeze the parameters of the base LLM. The layer parameters and the parameters of the SAE encoder are used to train only the final linear head weights. ; Step 3.2: Loss Function Calculation: Using Pairwise Preference Data ,in To win the reply, This is a failed response. The loss is calculated using the Bradley-Terry model: ; This represents all parameters of the reward model. The reward model is based on the question. With answer The reward score given.

[0039] By minimizing this loss, the model learns each interpretable feature. Contribution weight to human preferences (i.e., linear value head weights).

[0040] This method also includes: During the inference phase, for any input, the scalar reward output by the reward model is decomposed into a product of a set of activated sparse features and their corresponding weights to achieve feature-level attribution. Dynamic preference intervention can be achieved by directly modifying the weights of specific features without retraining the model.

[0041] Dynamic preference intervention specifically involves identifying a set of features that are positively correlated with the target preference, and multiplying the weights of the feature set by an amplification factor during inference to guide the model to give higher rewards to outputs that conform to the target preference.

[0042] Specifically, dynamic preference control based on feature weights In the inference phase, feature-level attribution means that for any input, the reward score output by SARM can be decomposed into a product of a set of activated features and their weights. By analyzing the activated features (using tools such as GPT-4 to interpret the meaning of the features), we can directly explain why the model gives a high or low score. For example, if the feature "aggressive language" has a negative weight, the model will penalize the input for containing aggressive content; if the feature "encouraging language" has a positive weight, the model will reward the input for containing encouraging content. Dynamic intervention eliminates the need to retrain the model; it works by directly modifying the weights of specific features. This can be used to intervene in model behavior. For example, to improve security, a set of features positively correlated with "security" can be identified, and the weights corresponding to these features can be artificially amplified during inference. (That is, multiplied by a coefficient). This will cause the model to give higher rewards to safe responses, thereby guiding safer behavior in downstream PPO training or Best-of-N sampling.

[0043] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. Feature-level interpretability: This invention utilizes a sparse autoencoder (SAE) to decouple the black-box reward signal into a linear combination of succinct features. Users can precisely see which specific concepts (such as "code quality," "polite tone," or "illegal suggestions") contribute to the reward score and their degree of contribution, solving the problem of opacity in traditional RM.

[0044] 2. No need for multidimensional labeled data: Unlike multidimensional reward models, SARM does not require expensive and highly subjective multidimensional (Helpfulness, Safety, etc.) rating data. It can be trained using only standard pairwise preference data (Win / Loss), significantly reducing data labeling costs.

[0045] 3. Dynamic Controllability: This invention allows for dynamic manipulation of model preferences during the inference phase by adjusting the weights of linear value heads. For example, the weights of the "security" feature can be instantly increased to pass security tests, or the weights of the "code" feature can be increased to optimize programming tasks, without retraining the entire model. Experiments show that adjusting the security feature weights can significantly shift the reward distribution of the target dataset to the right (better aligning with preferences) without affecting irrelevant samples.

[0046] 4. Superior performance: The experimental results are shown in Table 1. In the RewardBench2 benchmark test, the SARM-4B model built based on this invention achieved a total score of 73.6, which not only outperforms open source models with the same number of parameters, but also surpasses the Llama-3.1-Tulu-3 model with 70 parameters and some closed-source models (such as GPT-4.1).

[0047] Table 1: Reward Baseline Performance Score and Baseline Comparison

[0048] This invention, SARM, proposes a reward model construction method based on sparse autoencoder enhancement. First, it pre-trains the sequence-level hidden states of the intermediate layers of a basic language model using a general corpus. The sparse autoencoder (SAE) maps the originally opaque latent space into a sparse, unambiguous, and interpretable feature space. Then, the pre-trained SAE encoder is integrated into the reward model architecture, replacing subsequent layers of the basic model, and the encoder parameters are frozen. Next, linear value heads are used to weight and aggregate the extracted sparse features. Without requiring expensive multi-dimensional supervised annotations, the feature weights can be trained using only pairwise preference data through a standard loss function. Finally, the model not only outputs a scalar reward but also directly attributes the reward score to specific semantic features.

[0049] This invention can be applied simply and efficiently to various real-world scenarios, and is particularly suitable for reinforcement learning training of policy models under dynamic preferences and large-scale data filtering.

[0050] In training a policy model under dynamic preferences, this invention eliminates the need to retrain the reward model. It only requires adjusting the weights of specific features in the linear value head to change the reward signal in real time, guiding the policy model (e.g., PPO) towards new preferences (e.g., higher security or specific language styles). In large-scale data filtering, this invention utilizes interpretable feature scores to accurately identify and filter high-quality data from massive datasets that meet specific semantic requirements (e.g., logical rigor and harmlessness), which can then be used for fine-tuning or pre-training of downstream models.

[0051] Experiments on multiple datasets show that this invention is not only an effective method to improve the expressive power, discriminative power and accuracy of models, but also has extremely high flexibility.

[0052] This invention also provides an interpretable reward model product based on a sparse autoencoder, comprising: The basic language model module is used to extract semantic representations of the input sequence, its preceding... Layer parameters can be frozen or fine-tuned; The sequence-level sparse autoencoder module is integrated into the basic language model module. After the layer, it is used to transfer the first The hidden activation vector of the last word in the layer is mapped to a high-dimensional sparse feature vector, and its encoder parameters are frozen during the training phase of the reward model. The linear value head module, connected after the sparse autoencoder module, is used to weight and aggregate the sparse feature vectors into a scalar reward, the weights of which are learnable parameters obtained by training with preference data. Among them, the basic language model module number All original layers after the first layer are discarded.

[0053] Furthermore, the sparse autoencoder module is a TopK sparse autoencoder, and its encoding process is as follows: The decoding and reconstruction process is as follows: ;in and These are the encoding matrix and the decoding matrix, respectively. For bias, Given a sparse feature vector, the TopK operation is used to force the retention of the K features with the largest activation values.

[0054] like Figure 4 As shown, this embodiment of the invention also provides a training system for an interpretable reward model based on a sparse autoencoder, comprising: The pre-training unit 101 is used to perform unsupervised pre-training of the sequence-level sparse autoencoder using a general corpus, with the goal of minimizing the reconstruction error. Model building unit 102 is used to integrate the pre-trained sparse autoencoder into the base language model, discard subsequent layers, and add linear value headers; Preference training unit 103 is used to train the base language model by freezing the encoder parameters of the sparse autoencoder using pairwise preference data. The layer parameters and the weights of the linear head are optimized using a preference-learning loss function.

[0055] This invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the interpretable reward model construction method based on sparse autoencoders provided in the above embodiments.

[0056] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0057] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing an interpretable reward model based on a sparse autoencoder, characterized in that, Includes the following steps: Construct and pre-train a sequence-level sparse autoencoder, wherein the input sequence is fed into the pre-trained language model. The hidden activation vector of the last word is extracted as input, and the hidden activation vector is mapped to a high-dimensional sparse feature space through a sparse autoencoder to obtain a sparse feature vector. Unsupervised pre-training is then performed with the goal of minimizing the reconstruction error. Construct an interpretable reward model, in which the encoder part of a pre-trained sparse autoencoder is integrated into the base language model. After the first layer, discard the base language model. All original layers after the layer, and construct a linear value head on the sparse feature vector to weight and aggregate the sparse features into a scalar reward; A reward model is trained based on preference data, wherein the encoder parameters of the sparse autoencoder are frozen, and the base language model is trained using the pre-processor. The layer parameters and the weights of the linear value heads are optimized using a preference learning loss function with paired preference data, enabling the model to learn the weight of each interpretable feature's contribution to human preferences.

2. The method according to claim 1, characterized in that, The sparse autoencoder is a TopK sparse autoencoder, and its encoding process is represented as follows: ,in The parameter representing the pre-bias term in the sparse autoencoder. The hidden state vector; The decoding and reconstruction process is represented as follows: ； in and These are the encoding matrix and the decoding matrix, respectively. For bias, for A sparse feature vector is generated, and the TopK operation is used to force the retention of the K features with the largest activation values, while setting the rest to zero. The loss function for the reconstruction error is: , Let represent the hidden state vector after reconstruction by the sparse autoencoder.

3. The method according to claim 1, characterized in that, The formula for calculating the scalar reward by weighting and aggregating sparse features using the linear value head is as follows: ,in For the first The activation strength of a sparse feature The learnable weights are the corresponding features.

4. The method according to claim 1, characterized in that, The preference learning loss function is a pairwise ranking loss based on the Bradley-Terry model: ； in, This represents all parameters of the reward model. The reward model is based on the question. With answer The reward points given To win the reply, This is a reply indicating failure. This is the sigmoid function.

5. The method according to claim 1, characterized in that, Also includes: During the inference phase, for any input, the scalar reward output by the reward model is decomposed into a product of a set of activated sparse features and their corresponding weights to achieve feature-level attribution. Dynamic preference intervention can be achieved by directly modifying the weights of specific features without retraining the model.

6. The method according to claim 5, characterized in that, The dynamic preference intervention specifically involves: identifying a set of features that are positively correlated with the target preference, and multiplying the weights corresponding to the feature set by an amplification factor during inference to guide the model to give higher rewards to outputs that conform to the target preference.

7. An interpretable reward model product based on a sparse autoencoder, characterized in that, include: The basic language model module is used to extract semantic representations of the input sequence, its preceding... Layer parameters can be frozen or fine-tuned; The sequence-level sparse autoencoder module is integrated into the basic language model module. After the layer, it is used to transfer the first The hidden activation vector of the last word in the layer is mapped to a high-dimensional sparse feature vector, and its encoder parameters are frozen during the training phase of the reward model. The linear value head module, connected after the sparse autoencoder module, is used to weight and aggregate the sparse feature vectors into a scalar reward, the weights of which are learnable parameters obtained by training with preference data. Among them, the basic language model module number All original layers after the first layer are discarded.

8. The model product according to claim 7, characterized in that, The sparse autoencoder module is a TopK sparse autoencoder, and its encoding process is as follows: The decoding and reconstruction process is as follows: ;in and These are the encoding matrix and the decoding matrix, respectively. For bias, Given a sparse feature vector, the TopK operation is used to force the retention of the K features with the largest activation values.

9. A training system for an interpretable reward model based on a sparse autoencoder, characterized in that, include: The pre-training unit is used to perform unsupervised pre-training of sequence-level sparse autoencoders using a general corpus, with the goal of minimizing reconstruction error; The model building unit is used to integrate the pre-trained sparse autoencoder into the base language model, discard subsequent layers, and add linear value headers. Preference training units are used to train the base language model by freezing the encoder parameters of the sparse autoencoder using pairwise preference data. The layer parameters and the weights of the linear head are optimized using a preference-learning loss function.

10. A non-transitory computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the interpretable reward model construction method based on a sparse autoencoder as described in any one of claims 1 to 6.

Images