Direct preference optimization model construction method, device, electronic device and storage medium

By introducing the Unified DPO framework and hyperparameter settings in the direct preference optimization method, the problem of insufficient paired noise processing capabilities of DPO is solved, and more stable and high-performance model training is achieved in the noise environment, which improves the model's anti-noise ability.

CN118428324BActive Publication Date: 2024-11-19UNIV OF SCI & TECH OF CHINA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410888281.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-04
Publication Date
2024-11-19
Estimated Expiration
2044-07-04

Smart Images

  • Figure CN118428324B_ABST
    Figure CN118428324B_ABST
Patent Text Reader

Abstract

The present invention discloses a method, device, electronic device and storage medium for constructing a direct preference optimization model, the method comprising: the direct preference optimization model construction method comprising: obtaining a fine-tuned initial model; selecting a uDPO framework for the initial model and setting the hyperparameters of uDPO to obtain an objective function; generating a target model according to the objective function. By constructing the uDPO framework and introducing new hyperparameters in uDPO, fine-grained control of data reliability is achieved, thereby providing a more balanced exploration and utilization strategy in a noisy environment. Secondly, while maintaining the inherent robustness of DPO to point noise, uDPO significantly improves the model's ability to resist paired noise by optimizing the worst paired scenario. Compared with DPO, it can achieve significant performance improvement on both noisy and noise-free data sets, and has the ability to resist paired noise while having the ability of DPO to resist point noise.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and specifically to a method, device, electronic device and storage medium for constructing a direct preference optimization model. Background Art

[0002] In the field of machine learning, especially in the optimization and training of large-scale language models (LLMs), Direct Preference Optimization (DPO) is a key technology that aims to adjust the model through human preference feedback to better match human values ​​and preferences. The core idea of ​​DPO is to directly optimize the text or response generated by the model through pairwise comparison without explicitly learning a reward model.

[0003] However, the performance of the DPO method will be significantly affected when faced with noisy training data sets. Noise can be divided into two main categories: point noise and pairwise noise. Point noise refers to the poor quality of individual data points, which may contain irrelevant or incoherent information. This type of noise usually comes from errors or deficiencies in the data collection process. Pairwise noise involves incorrect associations between data pairs, leading to misjudgments of preference rankings. This noise may come from labeling errors or inconsistencies in subjective judgments.

[0004] The existing DPO method has a certain robustness in dealing with point noise, but it is not capable of dealing with paired noise. In practical applications, training data often inevitably contains noise, which requires the DPO method to not only perform well on noise-free data, but also maintain stability and accuracy in the presence of noise. Summary of the invention

[0005] In this embodiment, a direct preference optimization model construction method is provided to solve the problem of insufficient paired noise processing capability in related technologies.

[0006] In a first aspect, the present invention provides a method for constructing a direct preference optimization model, the method comprising:

[0007] Get the initial model for fine-tuning;

[0008] Selecting a uDPO framework for the initial model and setting hyperparameters of uDPO to obtain an objective function;

[0009] A target model is generated according to the target function.

[0010] In an optional embodiment, before obtaining the fine-tuned initial model, the process includes:

[0011] Obtain a training data set and an initial model, wherein the training data set includes user feedback and preference pairs, and select a large-scale language model as a basic architecture;

[0012] The training data set is preprocessed to obtain a triple data set.

[0013] In an optional embodiment, preprocessing the training data set to obtain a triple data set includes:

[0014] Train the training data set through the initial model to obtain triple data ,in, x Represents a user query, Indicates the answer selected by the user. Indicates the user's refusal answer;

[0015] The triple data constitutes a triple data set after data cleaning.

[0016] In an optional embodiment, obtaining a fine-tuned initial model includes:

[0017] Fine-tuning the initial model using a noise-free dataset;

[0018] The initial model is trained using positive samples in the noise-free data set and the MLE loss function to obtain the prediction performance of the noise-free data set corpus and the noise level of the initial model training data.

[0019] In an optional embodiment, the uDPO framework is selected for the initial model and the hyperparameters of the uDPO are set to obtain the objective function including:

[0020] Get the uDPO framework;

[0021] Set the hyperparameters of uDPO;

[0022] The objective function is obtained, and the expression of the objective function is as follows:

[0023] ;

[0024] in, is the log-likelihood objective function of the data points, is the Lagrangian factor or hyperparameter, Represents triple data.

[0025] In an optional embodiment, after generating the target model according to the target function, the following steps are also included:

[0026] According to the preference pairs in the training data set, the target model is trained by the loss function of uDPO;

[0027] Obtain a validation set and use it to evaluate the performance of the trained target model; adjust the value of the hyperparameter based on the performance evaluation results;

[0028] The target model is continuously trained using the training data set until a preset number of iterations or performance index is reached.

[0029] Compared with the prior art, the direct preference optimization model construction method of the present invention has the following beneficial effects:

[0030] By building the uDPO framework, new hyperparameters are introduced in uDPO , achieving fine-grained control over data reliability, thus providing a more balanced exploration and exploitation strategy in a noisy environment. Secondly, while maintaining the inherent robustness of DPO to point noise, uDPO significantly improves the model's ability to combat paired noise by optimizing the worst-case paired scenario. Compared to DPO, it can achieve significant performance improvements on both noisy and noise-free data sets, and has the ability to combat paired noise while having the ability of DPO to combat point noise.

[0031] In a second aspect, the present invention proposes a direct preference optimization model, which is constructed by the above-mentioned direct preference optimization model construction method.

[0032] In a third aspect, the present invention provides a model building device, comprising:

[0033] The acquisition module is used to obtain the initial model for fine-tuning;

[0034] A configuration module, used for selecting a uDPO framework for the initial model and setting hyperparameters of uDPO to obtain an objective function;

[0035] A construction module is used to generate a target model according to the target function.

[0036] In a fourth aspect, the present invention proposes an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the above-mentioned direct preference optimization model construction method.

[0037] In a fifth aspect, the present invention proposes a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the direct preference optimization model construction method described above is implemented.

[0038] Compared with the prior art, the beneficial effects of the direct preference optimization model, device, electronic device and storage medium of the present invention are the same as the direct preference optimization model construction method described in the first aspect, so they will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A flowchart of a method for constructing a direct preference optimization model in an embodiment of the present invention;

[0040] Figure 2 It is a structural block diagram of a device for constructing a direct preference optimization model in an embodiment of the present invention;

[0041] Figure 3 4 is a structural block diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0043] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with general skills in the technical field to which this application belongs. The words "one", "a", "the", "these" and the like in this application do not indicate a quantitative limitation, and they may be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone. Generally, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0044] In an embodiment of the present invention, a method for constructing a direct preference optimization model is provided. Figure 1 is a flow chart of the direct preference optimization model construction method of the present invention, such as Figure 1 As shown, the model building method includes:

[0045] S100, obtaining a fine-tuned initial model;

[0046] In this embodiment, the pythia-2.8B open source model is selected as the initial model, and all word embeddings, hidden layer weights, etc. are based on pythia-2.8B.

[0047] Before getting the initial model for fine-tuning include:

[0048] Obtain training datasets and initial models. The training datasets include user feedback and preference pairs. Select a large-scale language model (LLM) as the infrastructure.

[0049] The training data set is preprocessed to obtain a triplet data set.

[0050] Furthermore, the training data set is preprocessed to obtain a triple data set including:

[0051] Train the training data set through the initial model to obtain triple data ,in, x Represents a user query, represents the answer selected by the user (in this embodiment, expressing a "preference"), Indicates a user's rejection of the answer (expressed as "less preferred" in this embodiment).

[0052] Triple data After data cleaning, a triple data set is formed. By cleaning the data, obvious errors and inconsistencies can be removed. In this embodiment, the data is preliminarily cleaned with the help of rules, which can be operated manually to ensure the quality of the data.

[0053] Furthermore, obtaining the initial model for fine-tuning includes:

[0054] The initial model is fine-tuned using a noise-free dataset. It should be noted that the noise-free dataset includes noise-free high-quality data, and the initial model is preliminarily fine-tuned using the noise-free high-quality data, so that the initial model adapts to specific tasks and data distribution.

[0055] In this embodiment, a supervised fine-tuning technique is used. Specifically, the initial model is trained using positive samples in the noise-free dataset and the MLE loss function to obtain the prediction performance of the noise-free dataset corpus and the noise level of the initial model training data.

[0056] S200, select the uDPO (Unified DPO) framework for the initial model and set the hyperparameters of uDPO to obtain the objective function;

[0057] The uDPO framework in the present invention is derived from the DPO framework. In order to better express the core idea of ​​the present invention and facilitate understanding, the existing DPO technology is first described.

[0058] DPO is a direct preference optimization method, which expresses the preference for an answer through them, expressed as ,in and Respectively The specific formula for the optimal and unoptimal completion is:

[0059]

[0060] DPO technology is essentially a convex optimization, which cannot provide a deeper understanding of physical properties. The process of DPO optimal solution is a standard distributed robust optimization objective, as shown in the following formula:

[0061]

[0062] distributed is and the initial distribution Satisfy Divergence is a distance measure, and the robust radius Therefore, the DPO is robust to point noise by adjusting the robustness factor in different noise scenarios. for verification.

[0063] The above is a standard DRO objective function. Analyzing DPO from the perspective of DRO, we can see that there is point noise in the preference data, which leads to the strategy in the DPO objective function. Unreliable, so it needs to learn a better strategy Improve it and receive a The introduction of DRO also provides a robust factor Understanding: The best DPO Relative to It is monotonically decreasing and follows the following relationship:

[0064]

[0065] The above formula illustrates the robustness factor and the robustness radius The inverse correlation between The value of delineates the boundaries of the search space; that is, an increase in noise requires a larger search space and thus a larger robustness radius Therefore, the robustness factor It is not just a hyperparameter of a model, but an indicator of the degree of noise in the data.

[0066] Therefore, based on the existing DPO technology combined with the data noise robustness perspective, it has physical meaning and can be personalized according to the above formula.

[0067] In summary, the uDPO framework is derived from the standard DPO framework, that is, the uDPO framework is the same as the overall framework of the standard DPO, which is beneficial for the uDPO proposed in this application to maintain the DPO robustness to point noise and paired noise while enhancing its robustness to paired noise.

[0068] Specifically, the uDPO framework is selected for the initial model and the hyperparameters of uDPO are set, and the objective function is obtained including:

[0069] Get the uDPO framework; combined with the above, the uDPO framework is the same as the overall framework of the standard DPO.

[0070] Set the hyperparameters of uDPO; it should be noted that the hyperparameters include the Lagrangian factor and the regularization coefficient ;

[0071] The objective function is obtained. The expression of the objective function is as follows:

[0072] ;

[0073] in, is the log-likelihood objective function of the data points, is one of the Lagrangian factors, i.e., hyperparameters, Represents triple data; further, the log-likelihood objective function is as follows:

[0074]

[0075] This objective function determines the robustness of the uDPO to paired noise.

[0076] Furthermore, the gradient of positive and negative samples is analyzed by the following formula:

[0077]

[0078] in, Relative to the reference strategy , by strategy Implicitly learned reward function. In the uDPO framework, and The gradients corresponding to positive and negative samples are used to increase the "preference" and reduce the probability of "under-preference" The probability of It is a factor.

[0079] In order to make the uDPO based on DPO robust to paired noise, the above function The expression is: , which improves the strategy update mechanism by weighting the positive samples.

[0080] It should be noted that in this embodiment, the hyperparameter No need to tune repeatedly, set it to the default value 1, regularization coefficient A value of 0.1 can meet the requirements of most data sets. Experiments have shown that the above settings can achieve stable performance improvements under various noise levels.

[0081] S300: Generate a target model according to the target function.

[0082] After generating the target model according to the target function, the following steps are also included:

[0083] According to the preference pairs in the training data set, the target model is trained through the loss function of uDPO; by optimizing the worst paired scenario, the model's robustness to paired noise is enhanced.

[0084] Get the validation set and use it to evaluate the performance of the trained target model; adjust the hyperparameters based on the performance evaluation results; evaluate the performance of the target model on the validation set, including the performance of point noise and paired noise, get the evaluation results, and adjust the hyperparameters based on the evaluation results, such as hyperparameters and hyperparameters value to make the model achieve better performance.

[0085] Continue to train the target model using the training data set until the preset number of iterations or performance indicators are reached. The stability of the target model is ensured by iterating the training of the target model.

[0086] When the target model is iterated a certain number of times or reaches a certain performance index, the target model is verified through an independent test set to ensure that the model has good generalization ability. Furthermore, the trained target model can be deployed to practical applications, such as online recommendation systems, automatic reply systems, etc.

[0087] Through the above process, the uDPO model can be effectively trained in noisy datasets, improve the robustness of the model, and provide high-quality output in practical applications.

[0088] In order to verify the conclusion, DPO and its corresponding derivative models cDPO and IPO were used as control models, and the preference optimization dataset was selected for experiment. According to the preference optimization evaluation criteria, the dataset was divided into training set, validation set and test set, and the accuracy of positive and negative samples was used as the metric. In order to demonstrate the robustness of uDPO, the positive and negative samples were flipped in different proportions. The experimental results are shown in the following table:

[0089]

[0090] It can be seen from the above table that at any flipping ratio, all indicators of uDPO surpass the conventional DPO and its derivative loss function. The uDPO loss function has surpassed the conventional DPO and its derivative loss function by tolerating the noise data in paired samples.

[0091] The embodiment of the present invention also proposes a direct preference optimization model, which is constructed by the above direct preference optimization model construction method. It should be noted that the objective function of the direct preference optimization model is the objective function recorded in the above method embodiment.

[0092] The embodiment of the present invention further provides a model building device, which is used to implement the above method embodiment, and the description has been made no further. The terms "module", "unit", "subunit" and the like used below can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceivable.

[0093] like Figure 2 As shown, Figure 2 : is a structural block diagram of a model building device in the present invention, the device comprising:

[0094] An acquisition module 101 is used to acquire a fine-tuned initial model;

[0095] Configuration module 102 is used to select the uDPO framework for the initial model and set the hyperparameters of uDPO to obtain the objective function; it should be noted that the uDPO framework is the same as the standard DPO framework as a whole, and the difference is that the objective functions of uDPO and DPO are different.

[0096] The construction module 103 is used to generate a target model according to the target function.

[0097] The fine-tuned initial model is obtained through the acquisition module 101, the uDPO framework is obtained based on the DPO framework through the configuration module 102, and the hyperparameters of the uDPO framework are set to obtain the objective function; the target model is obtained according to the objective function through the construction module 103.

[0098] like Figure 3 As shown, an embodiment of the present invention further provides an electronic device, which includes a memory 630 and a processor 610, wherein the memory 630 stores at least one computer executable instruction, and the processor 610 can be configured to run the computer executable instruction, and when the computer executable instruction is run by the processor 610, the above method embodiment is implemented.

[0099] Specifically, in this embodiment, the electronic device includes a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call the computer executable instructions in the memory 630 to execute the following method:

[0100] Get the initial model for fine-tuning;

[0101] Select the uDPO framework for the initial model and set the hyperparameters of uDPO to obtain the objective function;

[0102] Generate a target model based on the target function.

[0103] The logic instructions in the memory 630 can be implemented in the form of software functional units and used as

[0104] When an independent product is sold or used, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

[0105] The embodiment of the present invention further provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above method embodiment is implemented to perform, specifically:

[0106] Get the initial model for fine-tuning;

[0107] Select the uDPO framework for the initial model and set the hyperparameters of uDPO to obtain the objective function;

[0108] Generate a target model based on the target function.

[0109] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing a direct preference optimization model, characterized in that: The model building method comprises: Obtain a fine-tuned initial model, wherein the initial model is a pythia-2.8B open source model; Selecting a uDPO framework for the initial model and setting the hyperparameters of the uDPO to obtain an objective function, the uDPO framework is the same as the standard DPO framework; Getting the initial model for fine-tuning includes: Fine-tuning the initial model using a noise-free dataset; The initial model is trained using positive samples in the noise-free data set and the MLE loss function to obtain the prediction performance of the noise-free data set corpus and the noise level of the initial model training data; The uDPO framework is selected for the initial model and the hyperparameters of uDPO are set, and the objective function is obtained including: Get the uDPO framework; Set the hyperparameters of uDPO; The objective function is obtained, and the expression of the objective function is as follows: ; in, is the log-likelihood objective function of the data points, is the Lagrangian factor or hyperparameter, Represents triple data; where, Represents a user query, Indicates the answer selected by the user. Indicates the user's refusal answer; The log-likelihood objective function of a data point is as follows: ; A target model is generated according to the target function.

2. The direct preference optimization model construction method according to claim 1, characterized in that: Before getting the initial model for fine-tuning include: Obtain a training data set and an initial model, wherein the training data set includes user feedback and preference pairs, and select a large-scale language model as a basic architecture; The training data set is preprocessed to obtain a triple data set.

3. The direct preference optimization model construction method according to claim 2, characterized in that: After the target model is generated according to the target function, the following steps are also included: According to the preference pairs in the training data set, the target model is trained by the loss function of uDPO; Obtain a validation set and use it to evaluate the performance of the trained target model; Adjust the values ​​of hyperparameters based on performance evaluation results; The target model is continuously trained using the training data set until a preset number of iterations or performance index is reached.

4. A direct preference optimization model product, characterized in that: The model product is constructed by the direct preference optimization model construction method described in any one of claims 1 to 3.

5. A model building device, characterized in that: include: The acquisition module is used to obtain the initial model for fine-tuning. The initial model is the pythia-2.8B open source model. The acquisition of the initial model for fine-tuning includes: Fine-tuning the initial model using a noise-free dataset; The initial model is trained using positive samples in the noise-free data set and the MLE loss function to obtain the prediction performance of the noise-free data set corpus and the noise level of the initial model training data; A configuration module is used to select a uDPO framework for the initial model and set the hyperparameters of the uDPO to obtain an objective function. The uDPO framework is the same as the standard DPO framework. The uDPO framework is selected for the initial model and the hyperparameters of uDPO are set, and the objective function is obtained including: Get the uDPO framework; Set the hyperparameters of uDPO; The objective function is obtained, and the expression of the objective function is as follows: ; in, is the log-likelihood objective function of the data points, is the Lagrangian factor or hyperparameter, Represents triple data; where, Represents a user query, Indicates the answer selected by the user. Indicates the user's refusal answer; The log-likelihood objective function of a data point is as follows: ; A construction module is used to generate a target model according to the target function.

6. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the direct preference optimization model construction method according to any one of claims 1 to 3.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the direct preference optimization model construction method according to any one of claims 1 to 3 is implemented.