Large model preference alignment method improved by using online synchronization strategy

Abstract

The invention provides a large model preference alignment method improved by using an online synchronization strategy, a new preference alignment process is provided based on a synchronization strategy optimization and online learning method, transformation is carried out from a training process, links such as synchronous sampling and online parameter updating are added, and the optimization of the large model preference alignment is realized. Reward model updating and data distribution alignment in the model training process are achieved, and finally the financial question and answer ability is effectively improved. On the basis of keeping the high efficiency of the original method, the distribution difference between the strategies can be effectively reduced, the optimization direction of the optimal strategy is ensured, meanwhile, the supervision information of preference data is updated approximately in real time, the reliability of the reward model in the high reward system is enhanced, the human preferences are better aligned, and the method is suitable for popularization and application. Therefore, the reliability and generalization in the financial service field are ensured.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a large model preference alignment method improved by using an online synchronization strategy. Background Art

[0002] With the rapid development and widespread application of large models in the field of artificial intelligence, large models have made breakthroughs in various tasks, and people have begun to pay attention to the potential problems that these models may bring, especially the value tendencies they contain in the decision-making process. To solve these problems, researchers have introduced alignment technology. Reinforcement Learning from Human Feedback (RLHF) based on human preferences is the key to integrating human intuitive signals into the field of large models. Since the proposal of ChatGPT, preference alignment methods have been widely used by various large model manufacturers. The general process is to score the model output by training the Reward Model (RM), and then use the reinforcement learning algorithm to feed back human preferences to the model, so as to achieve the ultimate goal of aligning human preferences.

[0003] In the offline preference alignment stage, represented by the Direct Preference Optimization (DPO) method, by directly defining the preference loss as a policy parameter-related function, the binary cross entropy loss target optimization model can be directly optimized based on offline preference data. Although direct preference optimization can greatly reduce the training cost, offline learning and asynchronous sampling strategies lead to stricter requirements on data quality, reduced generalization ability outside the distribution, and inability to overcome the degradation and uncontrollability of model capabilities caused by data distribution offset.

[0004] like Figure 1 As shown, Invention 1 is based on a large amount of training data in the field of mathematics, and a mathematical evaluation model is trained to establish its self-evaluation optimization mechanism. By introducing the evaluation model into the training process, additional learning signals are provided for the final result, thereby guiding the model to improve its ability in solving mathematical problems.

[0005] This method obtains a mathematical evaluation model through pre-training, and uses it to fine-tune the model through rejection sampling, thereby realizing the optimization process of supervised learning from the perspective of approximate synchronization strategy. In the RLHF stage, the evaluation model is continued to be used to screen and strengthen the corpus and answers with low correctness scores, and further improve the mathematical ability of the model in the training of preference alignment. However, the distribution offset problem of the model in the parameter update process is ignored, and the evaluation model is prone to inaccurate scoring in the later stage of offline training.

[0006] Based on this invention 2, Figure 2As shown in the figure, a multi-level task-oriented training method and system are proposed. First, the training data is hierarchically divided based on the BERT model, so as to make full use of the hierarchical structure and correlation in financial data. In the process of preference alignment, the data is trained layer by layer, and the hierarchical division is dynamically adjusted through result backtracking, so as to improve the model's adaptability to training data and enhance its ability under continuous training.

[0007] There are two main solutions to the problems of data offset and reward model failure in existing technologies: one is to divide the data into layers or use curriculum learning methods to guide the model, but this requires a lot of prior knowledge and preliminary preparation, has high requirements for data quality and distribution diversity, and lacks universality. The second is to focus on improving a single ability in a specific field and rely on expert models in a field. On the one hand, this method will lead to the degradation of general capabilities, and on the other hand, the cost of training an expert model alone is too high, the scope of application is limited, and it cannot fundamentally solve the problems of reward model failure and data distribution offset caused by offline learning and asynchronous strategies.

[0008] The general preference learning process is carried out in the offline stage for the sampling of preference data, which is mainly divided into two stages: training reward model and direct preference alignment. In the prior art, in the reinforcement learning stage of large models in the financial field, direct preference optimization provides an efficient RLHF training method, which does not require explicit fitting of reward models and directly learns from human preference data sets. However, the offline learning method leads to overfitting problems in limited data sets, and the generated strategy model usually performs poorly when facing out-of-distribution data. In addition, due to the offline update method, the above-mentioned improvements to the preference alignment method cannot solve the prediction error of the reward function in high reward responses, resulting in the problem that human preferences and reward model preferences cannot be aligned during the training process. In business practice, it is manifested as insufficient generalization ability for specific financial problems, and limited by existing labeled data, there may even be a decline in special capabilities.

[0009] Invention 1: Xu Yifan, Liu Xiao, Liu Xinghan, et al. Method for improving the mathematical ability of large language models using self-evaluation mechanism:, CN202410519110[P]2024;

[0010] Invention 2: Zhou Yi, Shi Wanghua, Lei Yi. Multi-level task-oriented financial intelligence analysis model

[0011] Training method and system:,CN110134881A[P].2024. Summary of the invention

[0012] The purpose of the technical solution of the present invention is to provide a method for the hierarchical learning of the model's knowledge, so that the model has a tendency to gradually merge knowledge at different stages during the training process of preference alignment, and has certain advantages in training efficiency and overall generalization ability, so as to solve the problem of continuous optimization of a large model in a specific field during the alignment stage.

[0013] The technical solution of the present invention provides a large model preference alignment method improved by online synchronization strategy, comprising the following steps:

[0014] Step 1: Obtain an initial offline preference dataset that has been manually annotated or labeled by a reward model. The initial offline preference dataset includes multiple preference data of two model answers that are selected and rejected under the same user instruction;

[0015] Step 2: Sample the initial offline preference dataset and train the negative log-likelihood loss through likelihood estimation to obtain the trained reward model;

[0016] Step 3: Sample the training data of the trained reward model according to the sorting results of the reward difference;

[0017] Step 4: Input the preference data corresponding to the model answer with a low reward model score into the same model with different generation parameter configuration combinations and multiple inputs into different model save points or models with different parameter configurations to obtain multiple new answers, and use the trained reward model to score and sort them, and obtain the two model answers with the largest distribution difference, which are selected and rejected, and form a preference alignment training data set with the input question;

[0018] Step 5: Take the instruction requirements and different answers as a single input to the paired preference model, continuously predict the probability of the next token, and perform gradient propagation according to the loss function, so as to continuously optimize in the direction of increasing the difference in the preference interval to obtain the preference learning model;

[0019] Step 6: Input the preference alignment training data set into the preference learning model to perform approximate optimal strategy mining of the main task model to obtain semantic information of the data, and maximize the KL divergence of the auxiliary task model with the main task model under fixed constraints to obtain the combined optimization target;

[0020] Step 7: Repeat step 4 according to the combined optimization goal, and mix the latest sampled problems with a preset percentage to generate online preference data;

[0021] Step 8: Repeat the sorting and bucketing operations in step 3 according to step 7, and mix them with the original data to form new preference data, so as to update the reward model;

[0022] Step 9: After repeating the training for the preset number of rounds, end the large model direct preference alignment training.

[0023] Preferably, the preference data includes two major categories: general and financial, and each major category is distinguished according to different category labels such as writing, dialogue, code, and problem solving.

[0024] Preferably, the data sampling according to the sorting results of the reward differences includes: bucketing the training data of the trained reward model according to different category labels such as writing, conversation, code, and problem solving, and sampling the data between different buckets according to the sorting results of 100%, 80%, 50%, 30%, 10% and the probability after softmax normalization.

[0025] Preferably, the post-training reward model is as follows:

[0026]

[0027] Among them, θ represents the reward model parameters, L RM represents the reward model training loss, σ represents the sigmoid function, x represents the user instruction, and a c 、a r They represent the two model answers that are selected and rejected respectively under the same user instruction x.

[0028] Preferably, the input of each piece of data used by the trained reward model includes a question, a model answer, and a preference pair label.

[0029] Preferably, the preference learning model is as follows

[0030]

[0031] Among them, L DPO represents the preference alignment loss function, π θ , π ref The table shows the direct preference optimization model and the supervised fine-tuning reference model involved in the training process, and β represents the penalty term for rejected answers.

[0032] Preferably, in the combined optimization objective, maximizing the reward value under the constraint of the penalty term is solved using the preference learning model.

[0033] The technical solution of the present invention proposes a large model preference alignment method improved by online synchronization strategy. Based on the method of synchronization strategy optimization and online learning, a new preference alignment process is proposed. The training process is modified to add synchronous sampling and online parameter update and other links, so as to realize the reward model update and data distribution alignment in the model training process, and finally effectively improve the financial question-answering ability. The technical solution of the present invention can effectively reduce the distribution difference between strategies on the basis of maintaining the efficiency of the original method, ensure the optimization direction of the optimal strategy, and update the supervision information of the preference data in near real time, thereby enhancing the reliability of the reward model in the high reward system and better aligning human preferences, thereby ensuring reliability and generalization in the field of financial business. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A flow chart of a method for improving the mathematical capabilities of a large language model using a self-evaluation mechanism in the prior art;

[0035] Figure 2 It is a flowchart of a multi-level task-oriented financial intelligence analysis model in the prior art;

[0036] Figure 3 A flowchart of a large model preference alignment method improved by online synchronization strategy provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0038] like Figure 3 As shown, the embodiment of the present invention provides a large model preference alignment method improved by online synchronization strategy, and proposes a more effective online synchronization strategy process, which can reflect the distribution changes of training data in real time to model generation, thereby improving the controllability and effectiveness of the entire training of the financial large model, and realizing the use of some offline data to approximate the ideal state of objective distribution without adding additional training loss or computing overhead. The large model preference alignment method is specifically divided into the following steps:

[0039] 1. Construction of preference data in the financial field.

[0040] According to the manual labeling or reward model labeling method, the initial offline preference dataset D(a c ,a r |x), use a c 、ar They represent the selected and rejected answers of the two models under the same user instruction x. The preference data is specifically divided into two categories: general and financial. Each category is distinguished by different category labels such as writing, dialogue, code, and problem solving.

[0041] 2. Construction of reward model.

[0042] Initialize the reward model parameters θ. Here you can use the model results after supervised fine-tuning or a model with larger parameters. In order to predict the preference reward value, you need to add a fully connected network layer, sample based on the preference data obtained in step 1, and train the negative log-likelihood loss through likelihood estimation:

[0043]

[0044] Here we mainly consider paired preference data, L RM Represents the reward model training loss, and σ represents the sigmoid function. The input of each data used by the reward model needs to include the question, the model answer, and the preference pair label.

[0045] 3. From the training data of the reward model, sample data according to the sorting results of the reward difference to construct the preference data for training. Divide the preference data into buckets according to the different data types under general and financial. The data between different buckets are sampled according to the sorting results of 100%, 80%, 50%, 30%, and 10%. The data in the bucket is sampled according to the probability after softmax normalization.

[0046]

[0047] 4. Synchronize strategy sampling to build a new preference data set.

[0048] The preference data is reconstructed using the questions obtained through sorting, bucketing and sampling in step 3. Here, the present invention provides two synchronous sampling methods that can be used for hybrid data learning. One is to use different generation parameter configuration combinations under the same model parameters, including topk, topp, and temperature coefficient. The three groups of parameter values ​​that can be used are [0.9, 0.6, 0.95], [0.75, 0.95, 0.85], [0.6, 0.85, 0.9]; the other is to sample different model saving points to generate eight answers, and select three groups with the largest score distribution variance to form candidate preference answers.

[0049] Specifically, the financial and general questions corresponding to the model answers with low reward model scores are input into different model save points or models with different parameter configurations multiple times to obtain multiple new answers, and the reward model constructed in step 2 is used to score and sort them. Finally, the two model answers with the largest distribution difference, which are selected and rejected, are obtained, which constitute the data set for the subsequent preference alignment training together with the input questions.

[0050] 5. Preference alignment training method for dual-model synchronous learning

[0051] Based on the data set obtained in step 4, the supervised alignment trained model is directly fine-tuned to obtain the final model. The loss function used in the training process is as follows. It mainly consists of two parts: the main model training loss and the auxiliary loss, which are used to learn the optimal strategy and randomness respectively.

[0052] 5.1 Direct Preference Learning

[0053] The single input of the paired preference model consists of a command requirement and two different responses {x, a c ,a r}, during the training process, the probability P(a c 〉a r |x,a c ,a r ), and perform gradient propagation according to the following loss function, so as to continuously optimize in the direction of increasing the difference in the preference interval.

[0054]

[0055] L DPO represents the preference alignment loss function, π θ , π ref The table shows the direct preference optimization model and the supervised fine-tuning reference model involved in the training process, with a c 、a r and σ are consistent with the meaning of the reward model, and β represents the penalty term for rejected answers.

[0056] 5.2 Dual-model synchronous learning

[0057] Based on 4, the dual-model synchronous learning system sets two learning tasks. The main task model continuously approaches the optimal strategy based on the direct preference alignment method of all historical data sampling, thereby fully mining the semantic information of the data. The auxiliary task model maximizes the KL divergence with the main task model under fixed constraints. The combined optimization objectives are as follows:

[0058]

[0059] in The optimization goal of the alignment stage is to maximize the reward value under the constraint of the penalty term, which can be solved using the direct preference alignment in 5.1 above.

[0060] π * , π t Represent the main task model and the auxiliary task model respectively, t represents the current training time step, and any one greater than 0 can be found to make the above objectives converge. The total loss function for dual-model synchronous learning is

[0061] L merge =λL main +(1-λ)L aux

[0062] L main represents the main model training loss, and the method of updating parameters is the same as L in 5.1 DPO The optimization goal remains the same. aux Denotes the auxiliary model training loss D KL (π * (·∣x t )‖π t (·∣x t )). λ and η are hyperparameters and can be [[0.3,0.1], [0.4,0.3], [0.2,0.4]].

[0063] 6. Update the reference model and save multiple intermediate model versions during direct preference alignment training. The update frequency is to synchronize parameters and copy model results once every two rounds, so as to ensure that the optimization direction of the training process is always consistent with expectations, avoid excessive distribution differences, and improve the stability and controllability of training.

[0064] 7. Repeat step 4 and mix in 20% of the latest sampled questions to generate online preference data.

[0065] 8. Based on manual labeling or the reward model, the generated online preference data is sorted and bucketed in step 3, and mixed with the original data to form new preference data, thereby updating the reward model.

[0066] 9. Repeat the above process for 1 to 3 rounds to complete the direct preference alignment training of the financial large model under the online synchronization strategy proposed in the present invention.

[0067] The key points and protection points of the embodiments of the present invention are as follows:

[0068] Based on the synchronization strategy and online learning method, the original direct finance large model alignment training process is improved, a dual-model synchronization learning system is proposed, and a complete set of training methods is given. In addition, two specific implementation methods of the synchronization strategy are given, namely different parameter configurations or model save points.

[0069] The advantages of the embodiments of the present invention are as follows:

[0070] 1. It can overcome the problems of distribution shift and reward model failure caused by offline direct preference training.

[0071] 2. When the quality of preference data is poor, the optimization direction of the preference alignment training process should be corrected in a timely manner.

[0072] 3. The sampling method of online learning and synchronization strategy makes the model training process more stable and controllable, and can achieve better training results.

Claims

1. A large model preference alignment method improved by online synchronization strategy, characterized in that: The following steps are involved: Step 1: Obtain an initial offline preference dataset that has been manually annotated or labeled by a reward model. The initial offline preference dataset includes multiple preference data of two model answers that are selected and rejected under the same user instruction; Step 2: Sample the initial offline preference dataset and train the negative log-likelihood loss through likelihood estimation to obtain the trained reward model; Step 3: Sample the training data of the trained reward model according to the sorting results of the reward difference; Step 4: Input the preference data corresponding to the model answer with a low reward model score into the same model with different generation parameter configuration combinations and multiple inputs into different model save points or models with different parameter configurations to obtain multiple new answers, and use the trained reward model to score and sort them, and obtain the two model answers with the largest distribution difference, which are selected and rejected, to form a preference alignment training data set with the input question; Step 5: Take the instruction requirements and different answers as a single input to the paired preference model, continuously predict the probability of the next token, and perform gradient propagation according to the loss function, so as to continuously optimize in the direction of increasing the difference in the preference interval to obtain the preference learning model; Step 6: Input the preference alignment training data set into the preference learning model to perform approximate optimal strategy mining of the main task model to obtain semantic information of the data, and maximize the KL divergence of the auxiliary task model with the main task model under fixed constraints to obtain the combined optimization target; Step 7: Repeat step 4 according to the combined optimization goal, and mix the latest sampled problems with a preset percentage to generate online preference data; Step 8: Repeat the sorting and bucketing operations in step 3 according to step 7, and mix them with the original data to form new preference data, so as to update the reward model; Step 9: After repeating the training for the preset number of rounds, end the large model direct preference alignment training.

2. The large model preference alignment method improved by online synchronization strategy as claimed in claim 1, characterized in that: The preference data includes two major categories: general and financial. Each major category is divided into different category labels such as writing, dialogue, code, and problem solving.

3. A large model preference alignment method improved by online synchronization strategy as claimed in claim 2, characterized in that: The data sampling according to the sorting results of the reward differences includes: bucketing the training data of the trained reward model according to different category labels such as writing, conversation, code, and problem solving, and sampling the data between different buckets according to the sorting results of 100%, 80%, 50%, 30%, 10% and the probability after softmax normalization.

4. The large model preference alignment method improved by online synchronization strategy as claimed in claim 1, characterized in that: The post-training reward model is as follows: Among them, θ represents the reward model parameters, L RM represents the reward model training loss, σ represents the sigmoid function, x represents the user instruction, and a c 、a r They represent the two model answers that are selected and rejected respectively under the same user instruction x.

5. The large model preference alignment method improved by online synchronization strategy as claimed in claim 1, characterized in that: The input of each piece of data used by the trained reward model includes a question, a model answer, and a preference pair label.

6. The large model preference alignment method improved by online synchronization strategy as claimed in claim 1, characterized in that: The preference learning model is as follows Among them, L DPO represents the preference alignment loss function, π θ , π ref The table shows the direct preference optimization model and the supervised fine-tuning reference model involved in the training process, and β represents the penalty term for rejected answers.

7. The large model preference alignment method improved by online synchronization strategy as claimed in claim 1, characterized in that: In the combined optimization objective, the preference learning model is used to solve the problem of maximizing the reward value under the constraint of the penalty term.

Images