Direct preference optimization method and device for sparse feature constraints

Abstract

The present application proposes a direct preference optimization method and device for sparse feature constraints, comprising the following steps: inputting each preference sample pair into a reference model of frozen parameters to obtain the corresponding reference dynamic marginal value and reference sparse activation feature; copying the parameters of the reference model to obtain a dynamic policy model, and inputting each preference sample pair into the dynamic policy model to obtain the corresponding policy sparse activation feature; constructing a mean square error as a sparse feature constraint based on the difference between the policy sparse activation feature and the reference sparse activation feature, and using the reference dynamic marginal value as the marginal value constraint; constructing a loss function based on the sparse feature constraint, the marginal value constraint and the reward function, and updating the parameters of the dynamic policy model based on the loss function to obtain a preference model, and optimizing the model's preference by introducing sparse feature constraints and marginal value constraints on the original DPO framework, thereby achieving a balance between efficiency and controllability.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a direct preference optimization method and device for sparse feature constraints. Background Art

[0002] In the development of artificial intelligence, making large language models (LLMs) consistent with human values ​​and actual goals is a key challenge. Existing technologies generally use post-training methods such as instruction tuning and alignment to make large language models more consistent with human values. Among them, reinforcement learning from human feedback (RLHF) has become a representative post-training technology, which aims to guide the model to generate valuable and useful outputs by integrating human feedback. Although RLHF has been successful, it involves complex mechanisms such as reward modeling and policy gradients, which lead to high computational costs with great training complexity.

[0003] In order to reduce the complexity of training, direct preference optimization (DPO) is regarded as a more efficient alternative. Unlike the representative method proximal policy optimization (PPO) in RLHF, DPO adjusts the output probability of the model directly based on human preferences, thereby reducing training complexity and computational cost. DPO is currently developing mainly in two directions. One is to optimize efficiency, such as SimPO, which eliminates the model's need for a reference model by using the average logarithmic probability of the sequence as implicit normalization, thereby reducing memory usage and computing requirements. However, the performance of DPO is very sensitive to the strength of constraints from the reference policy, and over-simplification will lead to uncontrollability. The second is to optimize controllability by introducing token rewards and serialized KL divergence to solve problems in language continuity, diversity, and stability, but it will consume a lot of computing resources.

[0004] In summary, most of the current direct preference optimization methods can only achieve improvements in one direction, but cannot strike a balance between efficiency and controllability, and cannot achieve model alignment training that is simple, efficient, stable and controllable. Summary of the invention

[0005] The embodiments of the present application provide a direct preference optimization method and device for sparse feature constraints, which optimizes the preference of the model by introducing sparse feature constraints and marginal value constraints on the original DPO framework, thereby achieving a balance between efficiency and controllability.

[0006] In a first aspect, an embodiment of the present application provides a direct preference optimization method for sparse feature constraints, the method comprising:

[0007] Obtaining at least one preference sample pair with a marked preference as a training sample;

[0008] Input each preferred sample pair into the reference model of the frozen parameters to obtain the corresponding reference dynamic marginal value and reference sparse activation feature;

[0009] Copy the parameters of the reference model to obtain the dynamic strategy model, and input each preference sample pair into the dynamic strategy model to obtain the corresponding strategy sparse activation features;

[0010] The mean square error is constructed based on the difference between the strategy sparse activation feature and the reference sparse activation feature as the sparse feature constraint, and the reference dynamic margin is used as the margin constraint;

[0011] A loss function is constructed based on sparse feature constraints, marginal value constraints and reward functions, and the parameters of the dynamic strategy model are updated based on the loss function.

[0012] In a second aspect, an embodiment of the present application provides a direct preference optimization device for sparse feature constraints, comprising:

[0013] An acquisition module, used for acquiring at least one preference sample pair marked with a preference as a training sample;

[0014] A calculation module is used to input each preference sample pair into a reference model of frozen parameters to obtain a corresponding reference dynamic marginal value and a reference sparse activation feature; copy the parameters of the reference model to obtain a dynamic strategy model, and input each preference sample pair into the dynamic strategy model to obtain a corresponding strategy sparse activation feature;

[0015] The constraint construction module constructs the mean square error of the difference between the strategy sparse activation feature and the reference sparse activation feature as the sparse feature constraint, and uses the reference dynamic margin value as the margin value constraint;

[0016] The parameter updating module constructs a loss function based on sparse feature constraints, marginal value constraints and reward functions, and updates the parameters of the dynamic strategy model based on the loss function.

[0017] In a third aspect, an embodiment of the present application provides a readable storage medium, in which a computer program is stored. The computer program includes a program code for controlling a process to execute a process, and the process includes a direct preference optimization method for sparse feature constraints.

[0018] The main contributions and innovations of the present invention are as follows:

[0019] The embodiment of the present application introduces sparse feature constraints and marginal value constraints on the original direct preference optimization framework to optimize the preference of the model, aiming to overcome the instability and high computational overhead of DPO and alignment training methods during the training process, and to achieve a balance between efficiency and controllability, avoiding both the efficiency challenges of controlling each input token like token-level regularization (TDPO), and the controllability side effects of over-simplifying the loss function and completely discarding the reference model (SimPO).

[0020] Specifically, this scheme innovatively introduces sparse autoencoders in the internal representation layer of the dynamic strategy model and the reference model to generate sparse and interpretable sparse features, and constructs sparse feature constraints based on the sparse features to replace the traditional KL divergence, achieving significant constraints on computational costs and further enhancing the controllability and interpretability of the alignment process, thereby making the training process of this scheme have outstanding performance in computation, memory efficiency and generalization ability.

[0021] In addition, the alignment training of the direct preference optimization method proposed in this solution is carried out in two stages. This solution separates the calculation of the reference model from the training process. By calculating the output of the reference model offline and caching it, the reference model can be released during the alignment training process to bring a small and acceptable I / O requirement; and this solution only needs to cache the top-k reference sparse features of the reference model. Compared with the traditional method (TDPO) that requires optimization of each token, the space requirement of the direct preference optimization method of this solution is greatly reduced. The details of one or more embodiments of the present application are presented in the following drawings and descriptions to make other features, purposes and advantages of the present application more concise and easy to understand. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0023] Figure 1 is a method flow chart of a direct preference optimization method for sparse feature constraints according to an embodiment of the present application;

[0024] Figure 2 is a structural block diagram of a sparse feature constrained direct preference optimization device according to an embodiment of the present application;

[0025] Figure 3 It is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0027] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0028] Embodiment 1

[0029] like Figure 1 As shown, the embodiment of the present application provides a direct preference optimization method for sparse feature constraints. By introducing sparse feature constraints and marginal value constraints, the present solution achieves a better balance between computational cost, controllability and alignment effect. Specifically, refer to Figure 1 , the method comprising:

[0030] Obtaining at least one preference sample pair with a marked preference as a training sample;

[0031] Input each preferred sample pair into the reference model of the frozen parameters to obtain the corresponding reference dynamic marginal value and reference sparse activation feature;

[0032] Copy the parameters of the reference model to obtain the dynamic strategy model, and input each preference sample pair into the dynamic strategy model to obtain the corresponding strategy sparse activation features;

[0033] The mean square error is constructed based on the difference between the strategy sparse activation feature and the reference sparse activation feature as the sparse feature constraint, and the reference dynamic margin is used as the margin constraint;

[0034] A loss function is constructed based on sparse feature constraints, marginal value constraints and reward functions, and the parameters of the dynamic strategy model are updated based on the loss function to obtain the preference model.

[0035] In the step of "obtaining at least one pair of preference samples with a preference label as training samples":

[0036] The preference sample pairs of this solution are data sets that annotate user preferences. This solution optimizes the alignment strategy model through the preference sample pairs to obtain a preference model that can output a preference model that satisfies user preferences. Specifically, each preference sample pair includes two groups of answers for the model under the same prompt, and one of the answers is marked with user preference.

[0037] Models that have not been trained for alignment may make up stories or be overly perfunctory when responding. These problems are not in line with human preferences. For example, when a user asks the model "How is the weather today?", the model will give two answers: one is "The weather is good today."; the other is "The weather on December 2 is sunny, with a minimum temperature of 8°C, a maximum temperature of 21°C, good air quality, a PM2.5 index of 50, a humidity of 34, and a south wind of level 2." Obviously, the second answer is more in line with human preferences and will make people more satisfied. Therefore, this solution manually annotates the second answer to show that it is more in line with human values. This solution also forms a preference sample pair with the question and the two answers. Subsequently, the preference optimization of the policy model is performed through the manually annotated preference samples, so that the answer of the policy model is more in line with human preferences.

[0038] In the step of “inputting each preferred sample pair into the reference model of the frozen parameters to obtain the corresponding reference dynamic marginal value and reference sparse activation feature”, the formula of the reference dynamic marginal value is expressed as:

[0039]

[0040] in, is the reference dynamic margin value, β is the coefficient that controls the intensity of the KL divergence penalty, Representation reference model Generate the best answer for the input x of the preference sample pair The probability of Representation reference model Generate a non-optimal answer for the input x of the preferred sample pair probability.

[0041] Furthermore, the reference dynamic margin value is a value obtained after normalizing the calculation length of each preference sample pair.

[0042] Specifically, referring to the dynamic margin value can help the model control the output text length in direct preference optimization, so that the model can focus on key preference differences to adjust parameters and improve performance. In addition, by referring to the dynamic margin value, length normalization can be performed to avoid misjudgment caused by preference sample pairs of different lengths, thereby ensuring that the evaluation of preference differences in direct preference optimization is objective and fair, and improving the stability and reliability of training.

[0043] In the step of "inputting each preference sample pair into the reference model of frozen parameters to obtain the corresponding reference dynamic marginal value and reference sparse activation feature", each preference sample pair is input into the reference model of frozen parameters, and a pre-trained sparse autoencoder is used in the reference model to obtain the reference sparse activation feature of each preference sample pair.

[0044] Specifically, when the reference model takes the preferred sample pair as input, the output representation of the reference model at the lth layer at position t is obtained by using the sparse autoencoder that comes with the reference model and is pre-trained as the reference sparse activation feature.

[0045] In order to further reduce the amount of memory, the first K features with activation values ​​from large to small are retained as reference sparse activation features.

[0046] In a specific embodiment, k is a preset number of features, and preferably, k is 50.

[0047] Specifically, in actual data processing, each preferred sample pair will include a large number of features, some of which contribute less to the model. This solution uses a sparse autoencoder to filter the features to reduce the dimension of the data and speed up the alignment and reasoning of subsequent models.

[0048] It should be emphasized that this scheme obtains the reference sparse activation features and reference dynamic margin values ​​from the reference model of the input frozen parameters for each preference sample pair in the offline stage, and in order to reduce the computational overhead of subsequent alignment training, this scheme stores the obtained reference dynamic margin values ​​and reference sparse activation features in the cache.

[0049] Specifically, since the reference model plays an important role in the subsequent sparse feature constraints and marginal value constraints, but introducing the reference model in the alignment training process will increase the computational complexity of the alignment training process, this solution separates the calculation of the reference model from the alignment training process, and only reads and caches the reference dynamic marginal values ​​and reference sparse activation features in the offline stage. Storing the reference dynamic marginal values ​​and reference sparse activation features in the cache can ensure that the reference model can be released when the policy model is aligned subsequently, and only the reference dynamic marginal values ​​and reference sparse activation features of each preferred sample need to be called in the cache, which reduces the computational burden of the computer and only brings a small and acceptable I / O requirement.

[0050] The space requirement for caching the reference dynamic margin value and the reference sparse activation feature in this scheme is also very small. Specifically, since the reference dynamic margin value is a scalar, it only requires O(N) space to store in the cache, where N is the number of preferred sample pairs. Compared with the TDPO method that optimizes each token, since this scheme only takes the top-k sparse features to calculate the reference sparse activation feature, the reference sparse activation feature only requires O(2⋅N⋅k) space, where K is the preset number of features and N is the number of preferred sample pairs.

[0051] In the step of "copying the parameters of the reference model to obtain the dynamic strategy model, and inputting each preference sample pair into the dynamic strategy model to obtain the corresponding strategy sparse activation features", the dynamic strategy model is obtained by completely copying the parameters of the reference model, and the strategy sparse activation features are obtained in exactly the same way as the reference sparse activation features. Correspondingly, each preference sample pair is input into the dynamic strategy model, and a pre-trained sparse autoencoder is used in the dynamic strategy model to obtain the dynamic coefficient activation features of each preference sample pair.

[0052] Specifically, when the dynamic strategy model takes the preference sample pair as input, the output representation of the lth layer of the dynamic strategy model at position t is obtained by using the dynamic strategy model's own and pre-trained sparse autoencoder as the strategy sparse activation feature. Furthermore, the first K features with activation values ​​from large to small are retained as the strategy sparse activation features.

[0053] In some specific embodiments, the top k features (top-k) with the output activation values ​​from large to small are pooled to calculate the average activation value to obtain the strategy sparse activation feature, and then the mean square error of the strategy sparse activation feature and the corresponding reference sparse activation feature is constructed to achieve efficient feature-level constraints.

[0054] In the step of "using the difference between the strategy sparse activation feature and the reference sparse activation feature to construct the mean square error as the sparse feature constraint", the calculation formula of the sparse feature constraint is expressed as:

[0055]

[0056] in, sparse activation features for the strategy, is the reference sparse activation feature, i is the feature, is the index set of the k features with the highest activation values.

[0057] In some specific embodiments, in the step of “constructing a loss function based on sparse feature constraints, marginal value constraints and reward functions”, the formula of the loss function is expressed as follows:

[0058] ;

[0059] in, is the reference dynamic margin value after length normalization, used to represent the margin value constraint, is a sparse feature constraint, is a dynamic strategy model. is the expectation, D is the set of preferred sample pairs, is the reference model, β is the coefficient that controls the intensity of the KL divergence penalty, is the sigmoid function, Represents a dynamic policy model Generate the best answer for the input x of the preference sample pair The probability of Represents a dynamic policy model Generate a non-optimal answer for the input x of the preferred sample pair probability.

[0060] It should be noted that this solution uses marginal value constraints to perform additional alignment on the output of the alignment strategy model. The marginal value constraints are directly loaded from the cache of the reference model in the offline phase, thereby ensuring the stability of the output generated by the reference model and the alignment strategy model, thereby improving the alignment accuracy. In addition, this solution finally integrates sparse feature constraints, marginal value constraints, and reward functions to construct an optimized loss function, and finally uses the loss function to dynamically update the parameters of the dynamic strategy model to obtain a preference model that can output a preference that meets human preferences.

[0061] This scheme first uses the reference model in the offline stage to obtain the reference dynamic marginal value and reference sparse activation features, and then copies the reference model to obtain the dynamic policy model and performs alignment training. It can be seen that the loss function of the dynamic policy model training is composed of the reward function, sparse feature constraints, and marginal value constraints.

[0062] It should be noted that this scheme introduces sparse feature constraints and marginal value constraints on the basis of the original DPO, so the part about the reward function inherits the original DPO framework. For ease of understanding, the following describes the process of deriving the objective function of this scheme based on the original DPO:

[0063] The DPO framework uses the Bradley-Terry (BT) model for pairwise comparison. By incorporating the reward function into the BT model and using negative log-likelihood, the objective function in DPO aims to achieve alignment with the reference model by minimizing the reverse KL divergence. Therefore, the loss function of the original DPO is:

[0064]

[0065] Where D represents the data set containing preference sample pairs. and are the preferred and less preferred output results respectively, σ(∙) is the sigmoid function, represents the reference model, represents the strategy model for alignment training, is the coefficient that controls the intensity of the KL divergence penalty, and E is the expectation.

[0066] In this scheme, the implicit reward difference in DPO is observed, so it can be decomposed, thereby introducing a reference dynamic marginal value based on the reference model as a marginal value constraint:

[0067]

[0068] At the same time, inspired by the sparsity feature, this scheme uses the sparse feature activation of sparse autoencoders to approximate the KL divergence. This scheme introduces the feature-level sparse autoencoder and uses the mean square error (MSE) to measure the difference between the current model and the reference model in the feature space to obtain the sparse feature constraint. Finally, the marginal value constraint and the sparse feature constraint are integrated into the loss function of DPO to obtain the loss function of this scheme.

[0069] In some specific embodiments, the direct preference optimization method of this scheme improves the win rate by more than 5 percentage points in the AlpacaEval-2 and Arena-Hard benchmarks, improves the score in MT-Bench by 0.5, and maintains excellent output diversity. By constraining the changes of these features during training, we can achieve a leading effect on TDPO while significantly reducing the computational cost. In addition, we also conduct detailed ablation experiments to show the stable performance of our method under different temperatures and SAE layer selections.

[0070] Embodiment 2

[0071] Based on the same idea, refer to Figure 2 , the application also proposes a direct preference optimization device for sparse feature constraints, including:

[0072] An acquisition module, used for acquiring at least one preference sample pair marked with a preference as a training sample;

[0073] A calculation module is used to input each preference sample pair into a reference model of frozen parameters to obtain a corresponding reference dynamic marginal value and a reference sparse activation feature; copy the parameters of the reference model to obtain a dynamic strategy model, and input each preference sample pair into the dynamic strategy model to obtain a corresponding strategy sparse activation feature;

[0074] The constraint construction module constructs the mean square error of the difference between the strategy sparse activation feature and the reference sparse activation feature as the sparse feature constraint, and uses the reference dynamic margin value as the margin value constraint;

[0075] The parameter updating module constructs a loss function based on sparse feature constraints, marginal value constraints and reward functions, and updates the parameters of the dynamic strategy model based on the loss function to obtain a preference model.

[0076] Embodiment 3

[0077] This embodiment also provides an electronic device, referring to Figure 3 , comprises a memory 404 and a processor 402, wherein the memory 404 stores a computer program, and the processor 402 is configured to run the computer program to execute the steps in any of the above method embodiments.

[0078] Specifically, the processor 402 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0079] Among them, the memory 404 may include a large capacity memory 404 for data or instructions. For example, but not limitation, the memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these. In appropriate cases, the memory 404 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 404 may be inside or outside the data processing device. In a specific embodiment, the memory 404 is a non-volatile memory. In a specific embodiment, the memory 404 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, PROM for short), an erasable PROM (Erasable Programmable Read-Only Memory, EPROM for short), an electrically erasable PROM (Electrically Erasable Programmable Read-Only Memory, EEPROM for short), an electrically alterable ROM (Electrically Alterable Read-Only Memory, EAROM for short) or a flash memory (FLASH) or a combination of two or more of these. In appropriate circumstances, the RAM may be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM may be a fast page mode dynamic random access memory 404 (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0080] The memory 404 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 402 .

[0081] The processor 402 implements any one of the sparse feature-constrained direct preference optimization methods in the above embodiments by reading and executing computer program instructions stored in the memory 404 .

[0082] Optionally, the electronic device may further include a transmission device 406 and an input / output device 408 , wherein the transmission device 406 is connected to the processor 402 , and the input / output device 408 is connected to the processor 402 .

[0083] The transmission device 406 can be used to receive or send data via a network. Specific examples of the above-mentioned network may include a wired or wireless network provided by a communication provider of the electronic device. In one example, the transmission device includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 406 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0084] The input / output device 408 is used to input or output information. In this embodiment, the input information may be a preference sample, etc., and the output information may be a value of a loss function, etc.

[0085] Optionally, in this embodiment, the processor 402 may be configured to perform the following steps through a computer program:

[0086] Obtaining at least one preference sample pair with a marked preference as a training sample;

[0087] Input each preferred sample pair into the reference model of the frozen parameters to obtain the corresponding reference dynamic marginal value and reference sparse activation feature;

[0088] Copy the parameters of the reference model to obtain the dynamic strategy model, and input each preference sample pair into the dynamic strategy model to obtain the corresponding strategy sparse activation features;

[0089] The mean square error is constructed based on the difference between the strategy sparse activation feature and the reference sparse activation feature as the sparse feature constraint, and the reference dynamic margin is used as the margin constraint;

[0090] A loss function is constructed based on sparse feature constraints, marginal value constraints and reward functions, and the parameters of the dynamic strategy model are updated based on the loss function.

[0091] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0092] In general, various embodiments may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects of the invention may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flow charts, or using some other graphical representation, it should be understood that, as non-limiting examples, the boxes, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0093] Embodiments of the present invention may be implemented by computer software that is executable by a data processor of a mobile device, such as in a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets and / or macros may be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. A computer program product may include one or more computer executable components configured to perform an embodiment when the program is run. One or more computer executable components may be at least one software code or a portion thereof. In addition, at this point, it should be noted that, for example, Figure 3 Any block of the logic flow in the program may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on physical media such as memory chips or storage blocks implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as, for example, DVDs and their data variants, CDs, etc. Physical media are non-transitory media.

[0094] Those skilled in the art should understand that the technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0095] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A direct preference optimization method for sparse feature constraints, characterized in that: Includes steps: Obtaining at least one preference sample pair with a marked preference as a training sample; Input each preferred sample pair into the reference model of the frozen parameters to obtain the corresponding reference dynamic marginal value and reference sparse activation feature; Copy the parameters of the reference model to obtain the dynamic strategy model, and input each preference sample pair into the dynamic strategy model to obtain the corresponding strategy sparse activation features; The mean square error is constructed by the difference between the strategy sparse activation feature and the reference sparse activation feature as the sparse feature constraint, and the reference dynamic margin value is used as the margin value constraint. The calculation formula of the sparse feature constraint is expressed as: in, sparse activation features for the strategy, is the reference sparse activation feature, i is the feature, is the index set of the k features with the highest activation values; The loss function is constructed based on the sparse feature constraint, marginal value constraint and reward function. The dynamic strategy model is updated based on the loss function to obtain the preference model. The loss function is expressed as follows: in, is the reference dynamic margin value after length normalization, used to represent the margin value constraint, is a sparse feature constraint, is a dynamic strategy model. is the expectation, D is the set of preferred sample pairs, is the reference model, β is the coefficient that controls the intensity of the KL divergence penalty, is the sigmoid function, Represents a dynamic policy model Generate the best answer for the input x of the preference sample pair The probability of Represents a dynamic policy model Generate a non-optimal answer for the input x of the preferred sample pair probability.

2. The direct preference optimization method for sparse feature constraints according to claim 1, characterized in that: Input each preferred sample pair into the reference model with frozen parameters, and use the pre-trained sparse autoencoder that comes with the reference model to obtain the output representation of the lth layer of the reference model at position t as the reference sparse activation feature; Each preference sample pair is input into the dynamic policy model, and the output representation of the lth layer of the dynamic policy model at position t is obtained as the policy sparse activation feature by using the pre-trained sparse autoencoder that comes with the dynamic policy model.

3. The direct preference optimization method for sparse feature constraints according to claim 2, characterized in that: The top K features with activation values ​​from large to small are retained as reference sparse activation features and / or strategy sparse activation features.

4. The direct preference optimization method for sparse feature constraints according to claim 3, characterized in that: The top K features with the output activation values ​​from large to small are pooled to calculate the average activation value to obtain the strategic sparse activation feature.

5. The direct preference optimization method for sparse feature constraints according to claim 1, characterized in that: The reference dynamic margin value is a value obtained by normalizing the calculation length of each preference sample pair.

6. The direct preference optimization method for sparse feature constraints according to claim 1, characterized in that: In the offline stage, a reference sparse activation feature and a reference dynamic margin value are obtained from a reference model of frozen parameters for each preference sample pair, and the reference dynamic margin value and the reference sparse activation feature are stored in a cache.

7. A direct preference optimization device for sparse feature constraints, characterized in that: include: An acquisition module, used for acquiring at least one preference sample pair marked with a preference as a training sample; A calculation module, used for inputting each preference sample pair into a reference model of frozen parameters to obtain a corresponding reference dynamic marginal value and a reference sparse activation feature; Copy the parameters of the reference model to obtain the dynamic strategy model, and input each preference sample pair into the dynamic strategy model to obtain the corresponding strategy sparse activation features; The constraint construction module uses the difference between the strategy sparse activation feature and the reference sparse activation feature to construct the mean square error as the sparse feature constraint, and uses the reference dynamic margin value as the margin value constraint. The calculation formula of the sparse feature constraint is expressed as: in, sparse activation features for the strategy, is the reference sparse activation feature, i is the feature, is the index set of the k features with the highest activation values; The parameter update module constructs a loss function based on sparse feature constraints, marginal value constraints and reward functions. Based on the loss function, the parameters of the dynamic strategy model are updated to obtain a preference model. The loss function is expressed as follows: in, is the reference dynamic margin value after length normalization, used to represent the margin value constraint, is a sparse feature constraint, is a dynamic strategy model. is the expectation, D is the set of preferred sample pairs, is the reference model, β is the coefficient that controls the intensity of the KL divergence penalty, is the sigmoid function, Represents a dynamic policy model Generate the best answer for the input x of the preference sample pair The probability of Represents a dynamic policy model Generate a non-optimal answer for the input x of the preferred sample pair probability.

8. A readable storage medium, characterized in that: The readable storage medium stores a computer program, wherein the computer program includes a program code for controlling a process to execute a process, wherein the process includes a direct preference optimization method for sparse feature constraints according to any one of claims 1-6.

Images