A model preference alignment method and device based on direct preference optimization

Through dynamic adjustment and data filtering optimization DPO method, the value sensitivity problem of LLMs output aligned with human preferences is solved, significantly improving the model alignment performance, and providing a more robust and adaptable training paradigm.

CN118569348BActive Publication Date: 2024-11-08UNIV OF SCI & TECH OF CHINA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410919603.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-10
Publication Date
2024-11-08
Estimated Expiration
2044-07-10

AI Technical Summary

Technical Problem

Large language models (LLMs) face the challenge of how to ensure output is consistent with human preferences and values ​​in practical applications. The existing technology ignores the combined effect of hyperparameter selection and paired data quality on model alignment performance, resulting in value sensitivity question.

Method used

By dynamically adjusting the value and combining data quality considerations, the direct preference optimization (DPO) method is optimized, and a guided data filtering method is introduced to reduce the impact of outliers on model training. Dynamic hyperparameter calculation and moving average method based on individual reward differences are adopted. Improve performance of model alignment with human preferences.

Benefits of technology

It significantly improves the performance of DPO, provides a more robust and adaptable paradigm, and provides better alignment effects for the training of LLMs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118569348B_ABST
    Figure CN118569348B_ABST
Patent Text Reader

Abstract

The present invention discloses a model preference alignment method and device based on direct preference optimization, wherein the model preference alignment method based on direct preference optimization comprises: obtaining data, the data comprising a preference data set, initializing a model using supervised fine-tuning of the preference data set, and obtaining a model convergence state; sampling a batch from the preference data set according to the model convergence state and calculating individual reward differences; sampling samples from the preference data set according to individual reward differences, calculating batch-level hyperparameters; calculating losses using a DPO loss function according to the batch-level hyperparameters and updating the model using a gradient update algorithm. In the present invention, #imgabs0# optimizes the performance of DPO by dynamically calibrating #imgabs1# values ​​and considering data quality. In addition, #imgabs2# also introduces a data filtering method guided by #imgabs3# to reduce the impact of outliers on model training, and #imgabs4# significantly improves the performance of DPO on a variety of models and data sets.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and more specifically to a model preference alignment method and device based on direct preference optimization. Background Art

[0002] With the rapid development of artificial intelligence technology, large language models (LLMs) have made remarkable achievements in understanding and generating natural language text. These models can perform various tasks such as text summarization, question answering, language translation, etc., which has greatly promoted the progress of the field of natural language processing (NLP).

[0003] However, LLMs face a key challenge in practical applications: how to ensure that the outputs of these models are consistent with human preferences and values. To address this problem, researchers have proposed a variety of model alignment strategies, including the Direct Preference Optimization (DPO) method. As an attractive method, DPO improves the efficiency of aligning the model with human preferences by simplifying the policy model training process and directly using preference data to guide model learning. The core idea of ​​DPO is to dynamically adjust the output probability of the model by comparing human-preferred samples with unpreferred samples, so that the model is more inclined to generate responses that meet human expectations. This process involves a key hyperparameter , which is used to balance the integration of the original model configuration with the new preference. However, most of the existing techniques ignore the hyperparameters The joint impact of the choice of and pairwise data quality on model alignment performance. Summary of the invention

[0004] In this embodiment, a model preference alignment method and device based on direct preference optimization are provided to optimize the existing direct preference optimization (DPO) method, aiming to solve the problem of DPO in processing paired data with different information amounts and qualities. value sensitivity issues and dynamically adjust values ​​to improve the performance of large language models (LLMs) that align with human preferences.

[0005] In a first aspect, an embodiment of the present invention provides a model preference alignment method based on direct preference optimization, and the model preference alignment method based on direct preference optimization includes:

[0006] Acquire data, the data including a preference data set, initialize a model using supervised fine-tuning of the preference data set, and acquire a model convergence state;

[0007] sampling a batch from the preference dataset according to the model convergence state and calculating individual reward differences;

[0008] Sampling samples from the preference dataset according to the individual reward differences, and calculating batch-level hyperparameters;

[0009] The loss is calculated using the DPO loss function according to the batch-level hyperparameters and the model is updated using the gradient update algorithm.

[0010] In an optional embodiment, acquiring data, the data including a preference data set, initializing a model using supervised fine-tuning of the preference data set, and acquiring a model convergence state includes:

[0011] Get the preferred dataset, batch size, constraint coefficient, selection ratio, scaling factor, and learning rate;

[0012] Initializing the model using supervised fine-tuning of the preference dataset;

[0013] Get the model convergence status.

[0014] In an optional embodiment, sampling a batch from the preference data set according to the model convergence state and calculating the individual reward difference includes:

[0015] When the model has not converged, sampling a batch from the preference data set;

[0016] Obtaining sample triplets in the batch, and obtaining individual reward differences based on the sample triplets;

[0017] A hyperparameter is assigned to each of the triples.

[0018] In an optional embodiment, sampling from the preference dataset according to the individual reward differences, and calculating batch-level hyperparameters includes:

[0019] Using a probability model to evaluate the importance of the individual reward differences of each triplet, and obtaining the relative importance of the samples;

[0020] Use the moving average method to estimate the standard deviation and update the standard deviation;

[0021] taking a number of samples according to said relative importance;

[0022] Compute batch-level hyperparameters.

[0023] In an optional embodiment, the standard deviation is used to update the probability model.

[0024] In an optional embodiment, the calculation batch level hyperparameters are defined as:

[0025]

[0026] In the formula, To control the hyperparameters Adjust the scale parameter, For individual reward differences, is the threshold estimated using the moving average method, represents the baseline hyperparameters inherent to DPO, Represents the sample mean.

[0027] In an optional embodiment, calculating the loss using the DPO loss function according to the batch-level hyperparameters and updating the model using the gradient update algorithm includes:

[0028] Get batch-level hyperparameters;

[0029] Substituting the batch-level hyperparameters into the DPO loss function to calculate the loss;

[0030] Update the model using the gradient update algorithm.

[0031] Compared with the prior art, the model preference alignment method based on direct preference optimization of the present invention has the following beneficial effects:

[0032] The present invention Through dynamic calibration The performance of DPO is optimized by taking into account the data quality. Also introduced Guided data filtering methods to reduce the impact of outliers on model training. Through empirical evaluation, We significantly improve the performance of DPO on a variety of models and datasets, and provide a more robust and adaptable paradigm for the training of LLMs.

[0033] In a second aspect, an embodiment of the present invention provides a model preference alignment device based on direct preference optimization, comprising:

[0034] An initialization module, used to obtain data, wherein the data includes a preference data set, initialize the model using supervised fine-tuning of the preference data set, and obtain a model convergence state;

[0035] a guided filtering module for sampling a batch from the preference dataset and calculating individual reward differences according to the model convergence state;

[0036] A batch calibration module, configured to sample from the preference dataset according to the individual reward differences and calculate batch-level hyperparameters;

[0037] The model update module is used to calculate the loss using the DPO loss function according to the batch-level hyperparameters and update the model using the gradient update algorithm.

[0038] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor, a communication interface, a memory and a bus, wherein the processor, the communication interface and the memory communicate with each other through the bus, and the processor can call logic instructions in the memory to execute the steps of the method provided in the first aspect.

[0039] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the model preference alignment method based on direct preference optimization as described in the first aspect.

[0040] Compared with the prior art, the beneficial effects of the model preference alignment device, electronic device and storage medium based on direct preference optimization of the present invention are the same as those of the model preference alignment method based on direct preference optimization described in the first aspect, so they will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 It is a flow chart of a model preference alignment method based on direct preference optimization in an embodiment of the present invention;

[0043] Figure 2 Schematic diagram of the difference between the positive and negative sample reward scores in an embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram of the win rate score of the Anthropic-HH dataset evaluated by GPT-4 in an embodiment of the present invention;

[0045] Figure 4 This is a schematic diagram of the winning rate score of the "TL;DR" summary data set evaluated by GPT-4 in an embodiment of the present invention;

[0046] Figure 5 It is a structural block diagram of a model preference alignment device based on direct preference optimization in an embodiment of the present invention;

[0047] Figure 6 1 is a structural block diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0049] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with general skills in the technical field to which this application belongs. The words "one", "a", "the", "these" and the like in this application do not indicate a quantitative limitation, and they may be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone. Generally, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0050] First, in order to better illustrate the technical solution of the present invention, the effect of paired data quality on hyperparameters is studied. The impact of the selection is explained as follows:

[0051] The training function of DPO is as follows:

[0052]

[0053] in, represents the output probability distribution; X represents the input prompt content (the query raised by humans), represents the positive answer (the response preferred by the human annotator), represents a positive response (a response that is the opposite of the one preferred by the human annotator).

[0054] We believe that the effectiveness of DPO depends on two factors: hyperparameters Selection and quality of paired data.

[0055] We conducted an experimental demonstration. Specifically, we used the Anthropic HH dataset, which contains about 170,000 conversations between humans and automatic assistants. In this dataset, the questions asked by humans are represented as , with two responses Pairing, where represents the responses preferred by human annotators, while is another response. It is worth noting that another response The information value is preserved, which minimizes the differences between response pairs in this dataset, classifying it as a low-gap dataset. To further explore the impact of data quality on DPO, a synthetic dataset, called the high-gap dataset, was constructed. This high-gap dataset is different from the low-gap dataset in that it introduces greater differences between responses. Specifically, the alternative responses is generated by the supervised fine-tuning (SFT) Pythia-2.8B model, and the preference response Keep it consistent with the original dataset. And combine the two datasets in a 50% ratio to create a mixed gap dataset to combine the characteristics of the low gap and high gap datasets.

[0056] The results of the study are hyperparameters The optimal value varies with data quality, reflecting different performance patterns across datasets. As can be observed, in low-gap paired data, smaller hyperparameters This is likely because the information content of this data allows for lower hyperparameters. Promotes more substantial updates, thus improving alignment accuracy. Conversely, for high-gap paired data, keep hyperparameters low Values ​​of 0 may lead to overfitting, which can severely disrupt the alignment process. Mixed gap datasets (a combination of low gap and high gap datasets) show more subtle performance patterns, suggesting the need for dynamic hyperparameters Calibrate the strategy to adapt to different data qualities. Therefore, stick to fixed hyperparameters Values, such as configuring hyperparameters at the global level , which may not be sufficient to cope with the dynamics and diversity of real-world datasets.

[0057] Combination Figure 2 As shown, Figure 2 is a schematic diagram of the difference between the positive and negative sample reward scores, Figure 2 As can be seen, the data set exhibits significant outliers; Figure 2In this paper, we use the Pythia-2.8B model to assess data quality by examining the distribution of reward differences for each triplet in the training samples of the Anthropic HH dataset (hereafter defined as “individual reward differences”).

[0058] The tails of the density plot extend beyond the highlighted percentage range, indicating the presence of data samples with significantly higher or lower reward differences. In particular, cases where positive samples have significantly higher rewards relative to negative samples indicate low information value, as these differences may not meaningfully contribute to the model's learning process. The opposite case suggests possible labeling errors. Both cases deviate from the expected reasonable distribution range and are therefore classified as outliers.

[0059] Through empirical analysis, it is emphasized that DPO has a great influence on hyperparameters. The sensitivity of the choice and the frequent occurrence of outliers. Therefore, determining the optimal hyperparameters The value requires careful consideration of the quality of the paired data while also addressing the impact of outliers.

[0060] Based on the above, a model preference alignment method based on direct preference optimization is provided in an embodiment of the present invention. Figure 1 is a flow chart of the model preference alignment method based on direct preference optimization of the present invention, such as Figure 1 As shown, the process includes the following steps:

[0061] S100, obtaining data, where the data includes a preference data set, initializing a model using supervised fine-tuning of the preference data set, and obtaining a model convergence state;

[0062] Specifically, the data includes preference data sets , using the preference dataset Supervised fine-tuning of the initialized model , and get the model The convergence states include:

[0063] Get the preference dataset , batch size , constraint coefficient , select the scale , scaling factor And the learning rate ;

[0064] Initialize the model using supervised fine-tuning on the preferred dataset ;

[0065] Get the model Convergence state.

[0066] The above operations are common steps for model preprocessing, so they will not be elaborated here.

[0067] S200, from the preferred data set according to the model convergence state Sample a batch And calculate the individual reward differences ;

[0068] It should be noted that sampling a batch refers to the process of randomly selecting a part of samples from the training data set for training.

[0069] Specifically, according to the model convergence status, the preferred dataset Sample a batch And calculate the individual reward difference include:

[0070] When the model does not converge, Sample a batch ;

[0071] Get Batch Sample triples It should be noted that the sample triple , based on the sample triples Get individual reward differences ;

[0072] It should be further explained that for individual reward differences , which represents the difference between the winning and losing rewards of a triplet, as a measure of the quality of the paired data, and the individual reward difference is defined as

[0073] For each triple Assigning Hyperparameters ;

[0074] Based on the guiding principle, a straightforward approach is to create a Assign a unique hyperparameter , let each hyperparameter becomes the custom parameter of its corresponding triple. This instance-level dynamic The adjustment can be formalized as follows:

[0075]

[0076] in, represents the inherent benchmark hyperparameter of DPO, usually set to 0.1, represents a preset threshold (or initial threshold), and the coefficient is a function that adjusts individual reward differences in the interval [0, 1]. right In particular, when = 0, Keep in unchanged, thereby maintaining the standard DPO framework without modification.

[0077] The above equation shows that and Monotonically increasing, allowing the model to adjust hyperparameters based on the difference in rewards between paired samples value.

[0078] It should be noted that such instance-level adjustments may introduce instability during training. Based on previous studies, small batch methods can help avoid saddle points or local minima and reduce the impact of noise. Drawing on these advantages, we provide a method for adjusting the hyperparameters. The batch-level dynamic estimation method is as follows:

[0079] S300: Sample from the preference data set according to individual reward differences and calculate batch-level hyperparameters ;

[0080] It should be noted that the calculation batch level hyperparameters are defined as:

[0081] ;

[0082] In the formula, To control the hyperparameters Adjust the scale parameter, For individual reward differences, is the threshold estimated by the moving average method, represents the baseline hyperparameters inherent to DPO, Represents the sample mean.

[0083] It should be noted that the threshold is estimated using the moving average method as follows:

[0084] ;

[0085] in, is a momentum coefficient. This batch-level calibration method introduces only one new parameter , used to control hyperparameters Adjust the scale. Calculate Directly in the DPO program and without additional computational overhead.

[0086] Specifically, samples are sampled from the preference dataset according to individual reward differences, and batch-level hyperparameters are calculated. include:

[0087] Use the probability model to classify each triple Individual reward differences Assess its importance and obtain the relative importance of samples;

[0088] It should be noted that, in combination with the above, in order to reduce the impact of outliers The adverse effects of the selection process introduce a Guided data filtering mechanism. Confidence rule, this strategy uses a probability model to determine the probability of each triple Individual reward differences To assess its importance, it is defined as follows:

[0089]

[0090] In the above formula, in order to reduce the training parameters in the actual process, and Represents the individual reward differences in the training data set The mean and standard deviation of .

[0091] The standard deviation is estimated using the moving average method and updated; the standard deviation is used to update the probability model.

[0092] Take a number of samples based on relative importance;

[0093] Compute batch-level hyperparameters.

[0094] Similar to We use the moving average method to dynamically estimate Values:

[0095]

[0096] This probability weighting assesses the relative importance of each sample and guides the selection samples (without replacement) based on their calculated probabilities .here, Indicates the selection ratio. The default value is 0.8. Preliminary experiments have shown that this value can optimize training efficiency and model accuracy.

[0097] This process is repeated in each training batch, ensuring that the training data is constantly updated to reflect the most informative samples, introducing hyperparameters Guided data filtering strategies are key to hardening the model against outliers, thus facilitating hyperparameter Accurate estimate of the value.

[0098] S400, calculate the loss using the DPO loss function according to the batch-level hyperparameters and update the model using the gradient update algorithm.

[0099] Specifically, the loss is calculated using the DPO loss function according to the batch-level hyperparameters and the model is updated using the gradient update algorithm, including:

[0100] Get batch-level hyperparameters;

[0101] Substitute the batch-level hyperparameters into the DPO loss function to calculate the loss:

[0102] It should be noted that the DPO loss function is:

[0103]

[0104] Update the model using the gradient update algorithm.

[0105] In this embodiment, the definition of calculating the gradient and updating the model is as follows:

[0106] ;

[0107] in, represents the model parameters corresponding to the iteration t-1 step, The model parameters corresponding to the iteration step t are Represents the training samples in the batch.

[0108] Return the final model .

[0109] In order to verify the conclusion, we first The improved DPO is empirically evaluated on two specific tasks: dialogue generation and summarization. The tasks are to compare the winning rate of the answers generated by the trained model with the answers generated by humans by fine-tuning a large model, with 50% being a tie with the answers generated by humans, and higher performance is better.

[0110] like Figure 3 and Figure 4 As shown, Figure 3 and Figure 4 The win rate analysis table for the GPT-4 evaluation shows that the win rate analysis of the GPT-4 evaluation of the one-step dialogue of the Anthropic HH dataset shows that Consistently excellent performance at all sampling temperatures, such as Figure 3 In addition, for the TL;DR summary task, It is the only method that achieves a winning rate of more than 50% at different sampling temperatures, such as Figure 4 The data also show that although the dynamic Both retrieval and data filtering enhance the effectiveness of DPO, but the effect of data filtering is particularly evident in the summarization task, which may be due to the inherently larger noise in the Reddit TL;DR summary dataset. It shows remarkable robustness to changes in sampling temperature. As the temperature gradually increases from 0.0 to 1.0, the win rate of the standard DPO drops to only 25%, while Maintained a relatively high level of performance, with a win rate of 54%.

[0111] The new DPO framework proposed in this paper is called , Through dynamic calibration The performance of DPO is optimized by taking into account the data quality. Also introduced Guided data filtering methods to reduce the impact of outliers on model training. Through empirical evaluation, We significantly improve the performance of DPO on a variety of models and datasets, and provide a more robust and adaptable paradigm for the training of LLMs.

[0112] The embodiment of the present invention also provides a model preference alignment device based on direct preference optimization, which is used to implement the above method embodiment, and will not be repeated hereafter. The terms "module", "unit", "subunit", etc. used below can implement a combination of software and / or hardware for predetermined functions. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware or a combination of software and hardware is also possible and conceivable.

[0113] like Figure 5 As shown, Figure 5 : is a structural block diagram of a model preference alignment device based on direct preference optimization in an embodiment of the present invention, the device comprising:

[0114] An initialization module, used to obtain data, wherein the data includes a preference data set, initialize the model using supervised fine-tuning of the preference data set, and obtain a model convergence state;

[0115] a guided filtering module for sampling a batch from the preference dataset and calculating individual reward differences according to the model convergence state;

[0116] A batch calibration module, configured to sample from the preference dataset according to the individual reward differences and calculate batch-level hyperparameters;

[0117] The model update module is used to calculate the loss using the DPO loss function according to the batch-level hyperparameters and update the model using the gradient update algorithm.

[0118] Figure 6 A structural block diagram of an electronic device provided by an embodiment of the present invention, such as Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the following method:

[0119] Acquire data, the data including a preference data set, initialize a model using supervised fine-tuning of the preference data set, and acquire a model convergence state;

[0120] sampling a batch from the preference dataset according to the model convergence state and calculating individual reward differences;

[0121] Sampling samples from the preference dataset according to the individual reward differences, and calculating batch-level hyperparameters;

[0122] The loss is calculated using the DPO loss function according to the batch-level hyperparameters and the model is updated using the gradient update algorithm.

[0123] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0124] An embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method provided by each of the above embodiments is implemented, for example, including: obtaining data, the data including a preference data set, initializing a model using supervised fine-tuning of the preference data set, and obtaining a model convergence state;

[0125] sampling a batch from the preference dataset according to the model convergence state and calculating individual reward differences;

[0126] Sampling samples from the preference dataset according to the individual reward differences, and calculating batch-level hyperparameters;

[0127] The loss is calculated using the DPO loss function according to the batch-level hyperparameters and the model is updated using the gradient update algorithm.

[0128] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0129] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A model preference alignment method based on direct preference optimization, used to improve the efficiency of aligning models with human preferences, characterized in that: The model preference alignment method based on direct preference optimization includes: Acquire data, wherein the data includes a preference dataset, wherein the preference dataset includes queries raised by humans, responses preferred by human annotators, and another response opposite to the responses preferred by human annotators, initialize a model using supervised fine-tuning of the preference dataset, and obtain a model convergence status; sampling a batch from the preference dataset according to the model convergence state and calculating individual reward differences; Sampling a batch from the preference data set according to the model convergence state and calculating individual reward differences includes: When the model has not converged, sampling a batch from the preference data set; Obtaining sample triplets in the batch, and obtaining individual reward differences based on the sample triplets; assigning a hyperparameter to each of the triples; Sampling samples from the preference dataset according to the individual reward differences, and calculating batch-level hyperparameters; Sampling samples from the preference dataset according to the individual reward differences and calculating batch-level hyperparameters include: Using a probability model to evaluate the importance of the individual reward differences of each triplet, and obtaining the relative importance of the samples; Use the moving average method to estimate the standard deviation and update the standard deviation; taking a number of samples according to said relative importance; Calculate batch-level hyperparameters; The computation batch level hyperparameters are defined as: In the formula, To control the hyperparameters Adjust the scale parameter, For individual reward differences, is the threshold estimated using the moving average method, represents the baseline hyperparameters inherent to DPO, represents the mean of the sample; The loss is calculated using the DPO loss function according to the batch-level hyperparameters and the model is updated using the gradient update algorithm.

2. The model preference alignment method based on direct preference optimization according to claim 1, characterized in that: Acquiring data, the data including a preference data set, initializing a model using supervised fine-tuning of the preference data set, and acquiring a model convergence state includes: Get the preferred dataset, batch size, constraint coefficient, selection ratio, scaling factor, and learning rate; Initializing the model using supervised fine-tuning of the preference dataset; Get the model convergence status.

3. The model preference alignment method based on direct preference optimization according to claim 1, characterized in that: The standard deviation is used to update the probability model.

4. The model preference alignment method based on direct preference optimization according to claim 1, characterized in that: Calculating the loss using the DPO loss function based on batch-level hyperparameters and updating the model using the gradient update algorithm include: Get batch-level hyperparameters; Substituting the batch-level hyperparameters into the DPO loss function to calculate the loss; Update the model using the gradient update algorithm.

5. A model preference alignment device based on direct preference optimization, used to improve the efficiency of aligning models with human preferences, characterized in that: include: an initialization module, configured to obtain data, wherein the data includes a preference data set, wherein the preference data set includes queries raised by humans, responses preferred by human annotators, and another response opposite to the responses preferred by human annotators, initialize the model using supervised fine-tuning of the preference data set, and obtain a convergence state of the model; a guided filtering module for sampling a batch from the preference dataset and calculating individual reward differences according to the model convergence state; Sampling a batch from the preference data set according to the model convergence state and calculating individual reward differences includes: When the model has not converged, sampling a batch from the preference data set; Obtaining sample triplets in the batch, and obtaining individual reward differences based on the sample triplets; assigning a hyperparameter to each of the triples; A batch calibration module, configured to sample from the preference dataset according to the individual reward differences and calculate batch-level hyperparameters; Sampling samples from the preference dataset according to the individual reward differences and calculating batch-level hyperparameters include: Using a probability model to evaluate the importance of the individual reward differences of each triplet, and obtaining the relative importance of the samples; Use the moving average method to estimate the standard deviation and update the standard deviation; taking a number of samples according to said relative importance; Calculate batch-level hyperparameters; The computation batch level hyperparameters are defined as: In the formula, To control the hyperparameters Adjust the scale parameter, For individual reward differences, is the threshold estimated using the moving average method, represents the baseline hyperparameters inherent to DPO, represents the mean of the sample; The model update module is used to calculate the loss using the DPO loss function according to the batch-level hyperparameters and update the model using the gradient update algorithm.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the model preference alignment method based on direct preference optimization as described in any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the model preference alignment method based on direct preference optimization as described in any one of claims 1 to 4 are implemented.