A multi-dimensional preference alignment method and system for large language models based on labeled reinforcement learning

Abstract

The present invention discloses a method and system for aligning multidimensional preferences of a large language model based on labeled reinforcement learning. The method of the present invention comprises: using multiple different reward models to score conversation sample data, obtaining sample preference confidence, performing labeling enhancement, and reconstructing the preference dataset; selecting conversation samples from the reconstructed dataset to perform supervised training on the large language model, and then performing confidence-based weighted direct preference optimization on the large language model; combining Pratt scaling to calibrate the large language model, iteratively updating the large language model parameters and calibration parameters, and realizing the optimal strategy for multidimensional human preference alignment. The present invention generates conversation sample confidence by calling multiple reward models as human agents, and combines labeled reinforcement learning to solve the shortcoming of the lack of multidimensional human preference features in mainstream alignment datasets, effectively improving the conversation quality of large language models, and has broad practical application prospects.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence natural language processing and relates to a large language model multi-dimensional preference alignment method and system based on labeled reinforcement learning. Background Art

[0002] With the rapid development of artificial intelligence (AI), large-scale language models have made significant progress in natural language processing. These models, trained on massive amounts of text data through unsupervised learning, are capable of generating high-quality text content. However, due to the completely unsupervised nature of their training, these models often struggle to accurately align with the multidimensional preferences of humans when generating content.

[0003] To improve the quality of language model responses, researchers have proposed large-scale language model alignment methods. These methods use question-answer pairs that reflect human preferences to help the model avoid generating responses that deviate from human preferences. Currently popular alignment methods include supervised fine-tuning, reinforcement learning optimization based on human feedback, and direct preference optimization. While these methods can help align models with human preferences to a certain extent, some challenges remain.

[0004] First, supervised fine-tuning has very high requirements for data quality. The existing preference datasets are relatively simply labeled and it is difficult to help the model avoid generating answers that deviate from human preferences, resulting in poor generalization performance. Second, the reinforcement learning method based on human feedback is very complex, requires a lot of computing resources, and the optimization process is unstable. Although direct preference optimization simplifies the alignment process, this method is prone to overfitting and can only learn fixed preferences, resulting in poor generalization performance. Finally, human preferences have diverse characteristics, and the above methods all rely on absolute preference question and answer datasets for alignment, which can only reflect relatively single human preferences and have significant limitations. Summary of the Invention

[0005] Purpose of the invention: To address the problem of single human preferences reflected in alignment objects in the prior art, the present invention provides a large language model multi-dimensional preference alignment method and system based on labeled reinforcement learning, so that the language model is closer to real human preferences and the conversation quality of the large model is improved.

[0006] Technical solution: To achieve the above-mentioned purpose, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for multi-dimensional preference alignment of a large language model based on labeled reinforcement learning, comprising the following steps:

[0008] Use multiple different reward models to score conversation sample data, obtain sample preference confidence, perform tag enhancement, and reconstruct the preference dataset;

[0009] After selecting dialogue samples from the reconstructed dataset to conduct supervised training on the large language model, the large language model is subjected to confidence-based weighted direct preference optimization.

[0010] Combined with Pratt scaling to calibrate the large language model, the large language model parameters and calibration parameters are iteratively updated.

[0011] Furthermore, for the same question in the preference dataset x , using multiple reward models for dialogue samples ( x,y w )and( x,y l ) to score and obtain the sample preference confidence:

[0012] ;

[0013] in, n is the number of reward models, y w To answer "chosen", y l To answer "rejected", is the indicator function, r i ( x,y w )and r i ( x,y l ) represent the i The reward model is used for dialogue samples ( x,y w )and( x,y l )’s score.

[0014] Furthermore, the reconstruction of the preference dataset includes: reselecting and recording the selected answers to the same question based on the confidence level; if the confidence level of the original "selected" answer calculated based on multiple reward models exceeds a set threshold, it remains unchanged; otherwise, the "selected" answer is exchanged with the "rejected" answer; and adding confidence features to the dataset.

[0015] Furthermore, the supervised training is to reselect the "selected" answers from the training set divided from the reconstructed data set to perform supervised training on the large language model. The alignment goal is to maximize the probability of the large language model generating the "selected" answer under the premise of a given question.

[0016] Furthermore, a direct preference optimization framework based on confidence weighting is used to obtain preference confidence through integrated calculation of multiple reward models. The probability ratio method based on the Bradley-Terry model is adopted to model human preferences, and an objective function that integrates multi-dimensional preference confidence is constructed to achieve alignment.

[0017] Furthermore, the objective function of the confidence-based weighted direct preference optimization is as follows:

[0018] ;

[0019] in, is the strategy model to be optimized, 、 Model Input x Generate answers y w 、 y l The probability of is the reference strategy model, 、 Model Input x Generate answers y w 、 y l The probability of is the training set of the reconstructed preference dataset, which contains triples ( x,y w ,y l ), is the temperature coefficient, the control strategy update amplitude, is the Sigmoid activation function, Expressing expectation.

[0020] Furthermore, in the large language model calibration combined with Pratt scaling, the calibration parameters are fitted using a calibration set divided from the reconstructed dataset, minimizing the negative log-likelihood loss on the calibration set:

[0021] ,

[0022] ,

[0023] ;

[0024] in, is the preference distribution output after model calibration, It's a model Input x Generate answersy w More likely than generating an answer y l The probability of the possibility, A and B are the calibration parameters to be fitted; initialize the calibration parameters A =1, B =0, and the calibration parameters and the large language model parameters are optimized alternately using the gradient descent algorithm.

[0025] In a second aspect, the present invention provides a large language model multi-dimensional preference alignment system based on labeled reinforcement learning, comprising:

[0026] The dataset reconstruction module is used to score the dialogue sample data using multiple different reward models, obtain the sample preference confidence, perform labeling enhancement, and reconstruct the preference dataset;

[0027] The model alignment module is used to select dialogue samples from the reconstructed dataset to perform supervised training on the large language model, and then perform confidence-based weighted direct preference optimization on the large language model.

[0028] The model calibration module is used to calibrate the large language model in combination with Pratt scaling, and iteratively update the large language model parameters and calibration parameters.

[0029] In a third aspect, the present invention provides a computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the multi-dimensional preference alignment method for a large language model based on labeled reinforcement learning are implemented.

[0030] In a fourth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the multi-dimensional preference alignment method for a large language model based on labeled reinforcement learning.

[0031] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0032] (1) The method of the present invention generates the confidence of dialogue samples by calling multiple reward models as human agents. Combined with labeled reinforcement learning, it solves the shortcomings of mainstream alignment datasets that lack multi-dimensional human preference features. It also improves the direct preference optimization method, improves the comprehensiveness, effectiveness and efficiency of the alignment process, and has broad application prospects in practical applications.

[0033] (2) The method of the present invention calibrates the language model after multi-dimensional preference alignment through Pratt scaling, and makes the language model closer to the real human preference distribution by calibrating the answer output probability of the strategy model, thereby obtaining a high-quality alignment model. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 Flowchart of the steps of an embodiment of the present invention.

[0035] Figure 2 This is a model architecture diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0037] An embodiment of the present invention discloses a multi-dimensional preference alignment method for a large language model based on labeled reinforcement learning, which mainly includes: using multiple different reward models to score dialogue sample data to obtain sample preference confidence, performing labeling enhancement, and reconstructing the preference dataset; selecting dialogue samples from the reconstructed dataset to perform supervised training on the large language model, and then performing confidence-based weighted direct preference optimization on the large language model; combining Pratt scaling to calibrate the large language model, and iteratively updating the large language model parameters and calibration parameters.

[0038] Mainstream large-scale model alignment algorithms usually rely on a single preference dataset, but human preferences are often multi-dimensional, such as security, relevance, and effectiveness. Such a single preference dataset will cause poor model alignment. This embodiment uses multiple reward models that focus on different preferences as human agents to perform multi-dimensional scoring for samples, obtain the preference confidence of the samples, and then perform the alignment process. Finally, Pratt scaling is introduced to further calibrate the language model to improve the quality of the model's answers. Figure 1 and Figure 2 As shown, this embodiment specifically includes the following steps:

[0039] S1: Dataset reconstruction phase: Input conversation samples from the preference dataset, score them according to multiple reward models focusing on different preferences, perform labeled reinforcement learning, and reconstruct the preference dataset after obtaining the confidence of the conversation samples.

[0040] Specifically, in the open source human preference dataset, for the same question x , there are two answers that reflect human preferences: the "selected" answer y w and "rejected" answer y l, considering the multidimensionality of human preferences, such as safety, effectiveness, relevance, etc., multiple reward models focusing on different preferences are used to evaluate dialogue samples ( x,y w )and( x,y l ) to score, and then get the confidence level:

[0041] ;

[0042] in, is the sample preference confidence, that is, the true preference distribution of humans, is the indicator function, r i ( x,y w )and r i ( x,y l ) represent the i The reward model is used for dialogue samples ( x,y w )and( x,y l ) score, a total of n The reward model enhances the original preference dataset by adding the confidence feature of the sample. Thus, the original dataset is reconstructed: first, the “selected” answer is reselected and recorded according to the confidence. If , the "selected" answer is still y w , otherwise its contents are compared with y l exchange; secondly, add sample preference confidence to the dataset to supplement the supervision information of human multi-dimensional preferences.

[0043] S2: Model alignment phase: The data reconstructed in step S1 is divided into a training set and a calibration set. High-quality dialogue samples reselected from the training dataset are used to conduct supervised training on the language model and perform preliminary alignment. Then, combined with the sample confidence, the training set is used to perform weighted direct preference optimization on the model. Specifically, it includes:

[0044] S21: Divide the data reconstructed in step S1 into a training set and a calibration set, and divide about 5% of the samples from the original training set as the calibration set , the remaining samples constitute the new training set , ensuring the independence of the calibration sets.

[0045] S22: Use the reselected “selected” answers from the training set y w Perform supervised training on the model and As the input of the language model, x i For the question part of the dialogue sample, y w_i The “selected” answers in the conversation sample are used. d The goal of this phase is to maximize the probability of the language model generating a "selected" answer given a question:

[0046] ;

[0047] in, is the strategy model to be optimized, m Yes y w length, Is the policy model for input x , generate the specified answer y k The supervised training can complete the alignment of the model with the expected behavior norms or criteria and obtain the model after preliminary alignment. , as a reference strategy model.

[0048] S23: Using the reward model as a human agent to simulate human preferences, the intuitive alignment goal of the model is:

[0049] ;

[0050] in, is a hyperparameter that balances the divergence value with the expected value of the reward model, r ( x,y ) is the reward model for question-answer pairs ( x,y )'s score, is the KL divergence function, Representation Policy Model For input x Generate answers y The probability of Representation Model For input x Generate answers y The probability of , combined with the Bradley-Terry probability model:

[0051] ;

[0052] in, 、 Represents the strategy model For input x Generate answers y1. y The probability of 2, 、 Represent the model For input x Generate answers y 1. y The probability of 2, is the temperature coefficient that controls the update frequency.

[0053] Combining the model's intuitive alignment goal with the Bradley-Terry probabilistic model, aligning language models based on human preference datasets The objective function is:

[0054] ;

[0055] in, is the target loss function, is the reference policy model obtained in step S22, x is the problem part of the sample, y w and y l are the responses of “selected” and “rejected” in the sample, 、 Model Input x Generate answers y w 、 y l The probability of 、 Model Input x Generate answers y w 、 y l The probability of is the Sigmoid activation function.

[0056] S24: Humans rarely have "black or white" preferences. To align with multi-dimensional human preferences, the sample preference confidence obtained by scoring according to multiple reward models in step S1 is converted to As the weight of direct preference optimization, then:

[0057] ;

[0058] Minimize the objective function to obtain a further aligned language model .

[0059] S3: Model calibration stage: Based on the model aligned in step S2, in order to make the model more in line with multi-dimensional human preferences, the model is calibrated in combination with Pratt scaling, and the model parameters and calibration parameters are iteratively updated.

[0060] Specifically, the Pratt scaling is introduced into the weighted direct preference optimization, and the calibration set obtained by the division in step S21 is used. , fit calibration parameters: For each sample in the calibration dataset, set the sample preference confidence As a proxy for the human-annotated preference probability, the distribution of preference labels in the calibration dataset is:

[0061] ,

[0062] ;

[0063] in, is the preference distribution output after model calibration, A and B are the calibration parameters to be fitted, is the model optimized in step S24 Input x Generate "selected" answers y w is more likely to generate a "rejected" response y l Minimize the negative log-likelihood loss on the calibration set:

[0064] ;

[0065] Among them, the initialization calibration parameters A =1, B =0, the calibration parameters and model parameters are optimized alternately using the gradient descent algorithm.

[0066] S4: Question-Answering Phase: Using the large language model calibrated in step S3, we conduct conversations on question examples in the test set and obtain high-quality answers that align with multi-dimensional human preferences.

[0067] In summary, the present invention discloses a multidimensional preference alignment method for large language models based on labeled reinforcement learning. It proposes the use of multiple reward models to reconstruct the preference dataset, and performs weighted direct preference optimization on the language model according to the sample confidence for comprehensive alignment. In addition, it proposes the use of Pratt scaling for model calibration, so that the language model is closer to the real human preference distribution, and ultimately achieves high-quality alignment with multidimensional human preferences.

[0068] The following experiment of multi-dimensional preference alignment of Qwen2.5-7B and Llama3-8B models on the public dataset HuggingFaceH4 / ultrafeedback_binarized is used to illustrate the advantages of the present invention over the prior art.

[0069] We used four open-source reward models to score the HuggingFaceH4 / ultrafeedback_binarized preference dataset, performed labeling enhancement, obtained sample preference confidence features, and reconstructed the dataset. We then used the alignment method disclosed in this paper to align and fine-tune the Llama3-8B and Qwen2.5-7B models. In this experimental setting, we selected four open-source reward models (RMs) with different preferences as human proxies and calculated the model output win rate for each reward model. In this experiment, we used all samples in the test set for win rate calculation. The experimental results are shown in Table 1. The four open-source reward models RM1, RM2, RM3, and RM4 are Skywork / Skywork-Reward-Llama-3.1-8B-v0.2, LxzGordon / URM-LLaMa-3.1-8B, Skywork / Skywork-Reward-Llama-3.1-8B, and RLHFlow / ArmoRM-Llama3-8B-v0.1, respectively. We compare the current mainstream alignment methods, including the instruction fine-tuning method (SFT), the human feedback-based reinforcement learning method (PPO), and the direct preference optimization method (DPO).

[0070] Table 1 Winning rate of the invented method based on various RMs on the UltraFeedback test set

[0071]

[0072] In Table 1, the baseline corresponds to the pre-trained Llama3-8B or Qwen2.5-7B basic model. The experimental data shows the proportion of answers to all questions in the test set that the strategy model optimized using the method of the present invention outperformed the strategy models optimized by other methods, as well as the win rate of the strategy model optimized by the method of the present invention relative to other strategy models. The win rate calculation formula of strategy model A relative to strategy model B is as follows:

[0073] ;

[0074] in, and Represents strategy model A and strategy model B respectively, M represents the number of test samples, Indicates that the strategy model A is for the test problem x m The answer given, Indicates the reward score of the question-answer pair, Represents the strategy model B for the test problem x m The answer given, Indicates the reward score of the question-answer pair, is the indicator function.

[0075] In addition, we calculated the normalized reward model score and used the AI large model Claude3.5-haiku for model evaluation. We randomly selected 500 test samples to calculate the model win rate. The experimental results are shown in Table 2.

[0076] Table 2 Normalized reward scores and AI evaluation win rates of experimental methods on the UltraFeedback dataset

[0077]

[0078] It can be seen that the experimental method in the present invention has achieved the best performance in all indicators, proving the feasibility of the invention motivation and invention method.

[0079] Based on the same inventive concept, an embodiment of the present invention also discloses a large language model multi-dimensional preference alignment system based on labeled reinforcement learning, including: a dataset reconstruction module, used to use multiple different reward models to score dialogue sample data, obtain sample preference confidence, perform labeling enhancement, and reconstruct the preference dataset; a model alignment module, used to select dialogue samples from the reconstructed dataset to perform supervised training on the large language model, and then perform confidence-based weighted direct preference optimization on the large language model; a model calibration module, used to calibrate the large language model in combination with Pratt scaling, and iteratively update the large language model parameters and calibration parameters.

[0080] An embodiment of the present invention also discloses a computer system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the multi-dimensional preference alignment method for a large language model based on labeled reinforcement learning are implemented.

[0081] An embodiment of the present invention also discloses a computer program product, including a computer program, which, when executed by a processor, implements the steps of the multi-dimensional preference alignment method for a large language model based on labeled reinforcement learning.

[0082] The program code for implementing the inventive method can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that the program code, when executed by the processor or controller, causes the steps of the inventive method to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as an independent software package and partially on a remote machine, or completely on a remote machine or server. The present invention is not described in detail herein, and all of these are known techniques to those skilled in the art.

[0083] It should be noted that the above content merely illustrates the technical idea of the present invention and cannot be used to limit the scope of protection of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications all fall within the scope of protection of the claims of the present invention.

Claims

1. A multi-dimensional preference alignment method for large language models based on labeled reinforcement learning, characterized by: The steps include: Use multiple different reward models to score conversation sample data, obtain sample preference confidence, perform tag enhancement, and reconstruct the preference dataset; After supervised training of the large language model using selected dialogue samples from the reconstructed dataset, the large language model is subjected to confidence-based weighted direct preference optimization. The objective function of the confidence-based weighted direct preference optimization is as follows: ； in, is the strategy model to be optimized, 、 Model Input x Generate answers 、 The probability of is the reference strategy model, 、 Model Input x Generate answers 、 The probability of is the training set divided by the reconstructed preference dataset, containing triples , is the temperature coefficient, the control strategy update amplitude, is the Sigmoid activation function, Expressing hope, is the sample preference confidence, n is the number of reward models, To answer "chosen", To answer "rejected", is the indicator function, and Respectively represent i Reward model for dialogue samples and 's scoring; Combined with Pratt scaling for large language model calibration, the large language model parameters and calibration parameters are iteratively updated; the calibration parameters are fitted using a calibration set partitioned from the reconstructed dataset, minimizing the negative log-likelihood loss on the calibration set: ， ， ； in, is the preference distribution output after model calibration, It's a model Input x Generate answers More likely than generating an answer The probability of the possibility, A and B are the calibration parameters to be fitted; initialize the calibration parameters A =1, B =0, and the calibration parameters and the large language model parameters are optimized alternately using the gradient descent algorithm.

2. The multi-dimensional preference alignment method for a large language model based on labeled reinforcement learning according to claim 1 is characterized in that: The reconstruction of the preference dataset includes: reselecting and recording the selected answers to the same question based on confidence; if the confidence of the original "selected" answer calculated based on multiple reward models exceeds a set threshold, it remains unchanged; otherwise, the "selected" answer is exchanged with the "rejected" answer; and adding confidence features to the dataset.

3. The multi-dimensional preference alignment method of a large language model based on labeled reinforcement learning according to claim 1 is characterized in that: The supervised training is to reselect the "selected" answers in the training set divided from the reconstructed data set to perform supervised training on the large language model. The alignment goal is to maximize the probability of the large language model generating the "selected" answer under the premise of a given question.

4. A large language model multi-dimensional preference alignment system based on labeled reinforcement learning, characterized by: include: The dataset reconstruction module is used to score the dialogue sample data using multiple different reward models, obtain the sample preference confidence, perform labeling enhancement, and reconstruct the preference dataset; The model alignment module is used to select dialogue samples from the reconstructed dataset to perform supervised training on the large language model, and then perform confidence-based weighted direct preference optimization on the large language model. The objective function of the confidence-based weighted direct preference optimization is as follows: ； in, is the strategy model to be optimized, 、 Model Input x Generate answers 、 The probability of is the reference strategy model, 、 Model Input x Generate answers 、 The probability of is the training set divided by the reconstructed preference dataset, containing triples , is the temperature coefficient, the control strategy update amplitude, is the Sigmoid activation function, Expressing hope, is the sample preference confidence, n is the number of reward models, To answer "chosen", To answer "rejected", is the indicator function, and Respectively represent i Reward model for dialogue samples and 's scoring; The model calibration module is used to calibrate the large language model in combination with Pratt scaling, and iteratively update the large language model parameters and calibration parameters. The calibration parameters are fitted using a calibration set partitioned from the reconstructed dataset, minimizing the negative log-likelihood loss on the calibration set: ， ， ； in, is the preference distribution output after model calibration, It's a model Input x Generate answers More likely than generating an answer The probability of the possibility, A and B are the calibration parameters to be fitted; initialize the calibration parameters A =1, B =0, and the calibration parameters and the large language model parameters are optimized alternately using the gradient descent algorithm.

5. A computer system comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the computer program is executed by a processor, the steps of the large language model multi-dimensional preference alignment method based on labeled reinforcement learning are implemented according to any one of claims 1 to 3.

6. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the large language model multi-dimensional preference alignment method based on labeled reinforcement learning are implemented according to any one of claims 1 to 3.

Images