A Preference Alignment Model Training Method Based on Restricted Preference Identification Sorting

Abstract

The present invention discloses a method for training a preference alignment model based on restricted preference identifier sorting, comprising: constructing a candidate preference data set based on an SFT data set; constructing a restricted preference identifier sorting data set based on the candidate preference data set; and training an SFT model using the restricted preference identifier sorting data set to obtain a preference alignment model. The present invention can avoid problems such as instability and overfitting in large language models during training, does not require manual annotation of a large amount of preference data, and greatly reduces the computing power required for training preference alignment models.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a preference alignment model training method based on restricted preference identification sorting. Background Art

[0002] After pretraining and supervised fine-tuning (SFT), the model still lacks in model security, advanced reasoning ability, and alignment of human value preferences. Large model preference alignment technology is a core method to improve the accuracy and usefulness of large models. After optimizing the SFT model based on reinforcement learning or direct preference optimization, it can effectively solve the model's security issues and greatly improve the model's advanced reasoning ability. It can be used to solve difficult problems, such as writing complex programs, solving complex mathematical problems, proving theorems, and multi-step analysis and induction of complex problems.

[0003] The current mainstream large-model preference alignment technologies include the reinforcement learning-based human return method (RLHF) and the non-reinforcement learning preference optimization algorithm. The reinforcement learning-based alignment algorithm uses the proximal policy optimization PPO for training, and needs to maintain multiple basic models such as the reference model, policy model, reward model, and value model at the same time. It consumes a lot of resources during training and is very inefficient. In order to solve the shortcomings of human feedback reinforcement learning based on PPO, the academic community has proposed many preference alignment algorithms based on non-reinforcement learning. These algorithms are all reward-free optimization strategies, that is, there is no need to train a reward model. Common non-reinforcement learning algorithms include the following:

[0004] 1. Direct Preference Optimization DPO (Direct Preference Optimization: Your Language Model is Secretly a Reward Model) designs an optimization function using the BT model by deriving the probability distribution under the optimal strategy setting. However, it has a serious overfitting problem, which will lead to a decrease in the score of positive preference responses.

[0005] 2. Preference ranking optimization RRHF (RRHF: Rank Responses to Align Language Models with Human Feedback without tears) and PRO (Preference Ranking Optimization for Human Alignment) prioritize responses from large models and use ranking-loss to optimize the normalized response probability distribution. However, this method itself uses off-policy, which leads to a large gap between the model output distribution and the ranking data distribution during optimization, greatly reducing the training stability and final effect. In addition, this method requires collecting a ranking preference list, which is very expensive in many scenarios.

[0006] 3. Contrastive Preference Optimization CPO (Contrastive Preference Learning: Learning from Human Feedback without RL) performs contrastive learning information entropy modeling on the model’s response probability. However, since the model’s negative samples come from preference data pairs, the number of data used to estimate preference rewards is insufficient, which can lead to severe overfitting results in the model similar to the DPO algorithm (Policy Optimization in RLHF: The Impact of Out-of-preference Data).

[0007] 4. The identity preference optimization algorithm IPO (A General Theoretical Paradigm to Understand Learning from Human Preferences) that emphasizes regular restricted mapping slightly alleviates the problems of the DPO algorithm such as reduced probability of adopting replies, decreased reward scores, and overfitting by explicitly transforming the constraints in the PPO algorithm. However, this optimization restriction does not fundamentally solve the inherent problems of this type of reward-free algorithm in optimizing the reply sequence.

[0008] 5. Optimization algorithms that do not require preference data pairs, such as KTO (Model Alignment as Prospect Theoretic Optimization), directly optimize the output performance of large models, and can optimize model preferences without providing partial order data pairs, which reduces the difficulty of data collection and improves data utilization efficiency. However, this type of algorithm requires the collection of a huge amount of binary preference labeled data to effectively model preferences, which makes it difficult to work in many scenarios.

[0009] In summary, current methods generally have three major problems: 1) A large amount of labeled preference data is required; 2) Training is unstable and prone to overfitting, resulting in model reward defects; 3) Online training requires a lot of computing resources. Summary of the invention

[0010] The object of the present invention is to provide a preference alignment model training method based on restricted preference identifier sorting to at least solve some of the above-mentioned technical problems.

[0011] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0012] A preference alignment model training method based on restricted preference identification sorting includes the following steps:

[0013] S1. Construct candidate preference dataset based on SFT dataset;

[0014] S2. Based on the candidate preference dataset, a restricted preference identification ranking dataset is constructed;

[0015] S3. Use the restricted preference identification sorting dataset to train the SFT model and obtain the preference alignment model.

[0016] Further, the S1 includes: S11, sampling N data pairs from the SFT data set, each data pair includes a question Q and an answer A; S12, based on the knowledge encyclopedia corpus text data set, generating a question set Q_{S} similar to each question Q by the SFT model, wherein the question set Q_{S} contains S similar sub-questions q, Q_{S}={q1, q2,.. ,..q N ,..q S}; S13. Retrieve a reference document associated with each question Q in the encyclopedia knowledge base. For each similar sub-question q of question Q, give m standard first answers, m second answers that are worse than the first answer, and m third answers that are better than the first answer according to the reference document; S14. Collect each similar sub-question q and the corresponding m first answers, m second answers, and m third answers into a list candidate preference data D'. Each question Q corresponds to S list candidate preference data D'. Collect the S list candidate preference data D' into a candidate preference data set D_{c}, D_{c}={D'1, D'2,..,..D' N ,..D' S}.

[0017] Further, the S2 includes: S21, based on the reference document, scoring all the first answers, second answers and third answers in each list candidate preference data D' of each question Q; S22, sampling k answers from each list candidate preference data D', k≤3m, assigning a preference identifier to each sampled answer, the preference identifier is <|Option:{number}|>, {number} is a number from 1 to k, and outputting the preference identifier corresponding to the highest-scoring answer in each list candidate preference data D'; S23, repeating S22 K times, and grouping the preference identifiers corresponding to the highest-scoring answers output each time into a restricted preference identifier sorting data P', and grouping the S restricted preference identifier sorting data P' of each question Q into a restricted preference identifier sorting data set P_{c}, P_{c}={P'1, P'2,.. ,..P' N ,..P' S}.

[0018] Furthermore, in S22, the k sampled answers are randomly shuffled, and then a preference identifier is assigned to each shuffled answer.

[0019] Furthermore, the scoring is based on a 1-10 point system, which includes five score ranges: 1-2 points: indicating that the answer is not available, 3-4 points: indicating that the answer meets the instruction requirements, 5-6 points: indicating that the answer is available, 7-8 points: indicating that the answer is good, and 9-10 points: indicating that the answer is perfect.

[0020] Furthermore, in S22, each list candidate preference data D' samples the same number of answers in five score segments of 1-2, 3-4, ..., 9-10.

[0021] Furthermore, the S3 includes: S31, sorting the data set according to restricted preference identifiers, and obtaining the output probability of each preference identifier in each list candidate preference data D' of each question Q; S32, calculating the loss between the preference pairs formed by any two preference identifiers in each list candidate preference data D' according to the output probability of the preference identifier, and adding the losses between all preference pairs to obtain the identifier partial order prediction preference strategy optimization loss; S33, calculating the probability score of the highest-scoring answer in each list candidate preference data D', optimizing the probability score of the SFT model in the current answer, and accumulating the probability scores of the highest-scoring answers in the S list candidate preference data D' to obtain the KL divergence; S34, adding the identifier partial order prediction preference strategy optimization loss and the KL divergence to obtain the final loss function; S35, training the SFT model based on the final loss function to obtain a preference alignment model.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] The present invention can avoid problems such as instability and overfitting in large language models during training, eliminates the need for manual labeling of large amounts of preference data, and significantly reduces the computing power required for training preference alignment models. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0025] Terminology explanation:

[0026] The ‌‌SFT model is ‌Supervised Fine-Tuning, a common deep learning strategy that is often used on pre-trained large language models.

[0027] ‌KL divergence (Kullback-Leibler Divergence), also known as relative entropy divergence.

[0028] Prompt is a technology based on artificial intelligence (AI) instructions that guide the output of language models through clear and specific instructions.

[0029] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] like Figure 1 As shown, the present invention provides a preference alignment model training method based on restricted preference identifier sorting, comprising the following steps:

[0031] S1. Construct candidate preference dataset based on SFT dataset;

[0032] S2. Based on the candidate preference dataset, a restricted preference identification ranking dataset is constructed;

[0033] S3. Use the restricted preference identification sorting dataset to train the SFT model and obtain the preference alignment model.

[0034] The present invention can avoid problems such as instability and overfitting in large language models during training, eliminates the need for manual labeling of large amounts of preference data, and significantly reduces the computing power required for training preference alignment models.

[0035] The S1 is to convert the SFT data set into a preference pair data set, and to generate diversified preference responses through restricted instruction evolution and document enhancement. Preferably, the S1 includes: S11, sampling N data pairs from the SFT data set, each data pair includes a question Q and an answer A; S12, based on the knowledge encyclopedia corpus text data set, generating a question set Q_{S} similar to each question Q by the SFT model, a question set Q_{S} contains S similar sub-questions q, Q_{S}={q1, q2,..,..q N ,..q S}; S13. Retrieve a reference document associated with each question Q in the encyclopedia knowledge base. For each similar sub-question q of question Q, give m standard first answers, m second answers that are worse than the first answer, and m third answers that are better than the first answer according to the reference document; S14. Collect each similar sub-question q and the corresponding m first answers, m second answers, and m third answers into a list candidate preference data D'. Each question Q corresponds to S list candidate preference data D'. Collect the S list candidate preference data D' into a candidate preference data set D_{c}, D_{c}={D'1, D'2,..,..D' N ,..D' S}.

[0036] By presetting a processing document prompt, a reference document associated with each question Q is retrieved from the encyclopedia knowledge base. The following is a detailed description in a specific scenario:

[0037] #Role Settings

[0038] Role X is an expert at information aggregation. Role X needs to aggregate valid information from multiple documents based on user questions.

[0039] #Task Description

[0040] Role X will see multiple documents, each corresponding to a brief question.

[0041] Role X first extracts the main information of the corresponding document based on this brief question and forms a summary description of the document of about 500 words.

[0042] Repeat the above operation for each document to form a summary information list.

[0043] Role X then needs to summarize this summary information based on user questions to form the final document.

[0044] Ensure that: the document summary must strictly come from the original document and cannot be fabricated; when summarizing summary information, make choices based on the problem and eliminate irrelevant and redundant information.

[0045] # Output requirements

[0046] Output a document containing the information required for the question, no more than 1,000 words.

[0047] # Input as follows:

[0048] ##User question: <User question content>

[0049] ## Documentation and brief list of issues

[0050] Question Q1: <Question content>

[0051] Document 1: <Document Content>

[0052] Question Q2: <Question content>

[0053] Document 2: <Document Content>

[0054] Question Q3: <Question content>

[0055] Document 3: <Document Content> ....

[0056] A set of questions Q_{S} similar to each question Q is generated by presetting a prompt for generating a candidate set of similar questions. The following is a detailed description in a specific scenario:

[0057] #Role Settings

[0058] Role X is an expert in analyzing user questions and instructions, and generates similar questions based on user questions and related documents in the knowledge encyclopedia corpus text dataset.

[0059] #Task Description

[0060] Role X needs to refer to the user's problem description, analyze its instruction characteristics and information processing requirements, find some entities, events, and relationships from the documents in the knowledge encyclopedia corpus text dataset, and organize them into similar questioning methods.

[0061] Role X needs to conduct similar analysis and problem construction from the following three types:

[0062] 1) Extended entities: Role X can find multiple different entities from the documents in the knowledge encyclopedia corpus text dataset and construct them using the user's question-asking method;

[0063] 2) Complicate the instruction requirements, complicate the instructions required by users, and add more output restrictions and requirements;

[0064] 3) Entity relationship questions: select the two entities with the most complex relationship described in the documents of the knowledge encyclopedia corpus text dataset, or mix 3 to 10 related entities to construct questions that reflect their relationship;

[0065] # Output requirements

[0066] Output a json list containing {{S}} similar questions:

[0067] {

[0068] "similar-questions": [<similar questions of type 1>, <similar questions of type 2>, <similar questions of type 3>]

[0069] }

[0070] # Input as follows:

[0071] ##User question: <User question content Q>

[0072] ##Reference document: <Document content>

[0073] ##Reference answer: <Answer content A>.

[0074] The prompt generated by presetting a preferred answer gives m standard first answers, m second answers that are worse than the first answer, and m third answers that are better than the first answer according to the reference document. The following is a detailed description in a specific scenario:

[0075] #Role Settings

[0076] Role X is an answer modifier, and can adjust the answer to make it better or worse according to needs.

[0077] #Task Description

[0078] First, role X needs to answer user questions based on reference documents and give a relatively standard answer.

[0079] Then, character X needs to modify the answer and construct a worse answer:

[0080] 1) Ignore some user instructions but keep the overall description clear;

[0081] 2) Some factual errors in the construction;

[0082] 3) Make the answer not smooth;

[0083] Finally, role X needs to modify the standard answer and construct a better answer:

[0084] 1) Polish your reply style to make it concise and clearly structured;

[0085] 2) Enrich the details of the command answers, organize the command results, and make the responses more reliable;

[0086] 3) Refer to the document to check if there are any errors in the answer and correct them;

[0087] 4) Reference documents, supplementary sections describe overly simple details.

[0088] 5) Adjust the overall flow of the answer to improve writing style and readability.

[0089] # Output requirements

[0090] Output a json list containing m groups of questions, each group of questions is as follows:

[0091] {

[0092] "standard-question": "<answer content>",

[0093] "worse-question":""<The answer is poor>"",

[0094] "good-question":""<The answer is good>""

[0095] }

[0096] # Input as follows:

[0097] ##User question: <User question content>

[0098] ##Reference document: <Reference document content>.

[0099] The S2 is responsible for generating a preference identifier for a specified content space. Preferably, the S2 includes: S21, based on the reference document, scoring all the first answers, second answers and third answers in each list candidate preference data D' of each question Q; S22, sampling k answers from each list candidate preference data D', k≤3m, assigning a preference identifier to each sampled answer, the preference identifier is <|Option:{number}|>, {number} is a number from 1 to k, and outputting the preference identifier corresponding to the highest-scoring answer in each list candidate preference data D'; S23, repeating S22 for K times, and grouping the preference identifiers corresponding to the highest-scoring answers output each time into a restricted preference identifier sorting data set P', and grouping the S restricted preference identifier sorting data sets P' of each question Q into a restricted preference identifier sorting data set P_{c}, P_{c}={P'1, P'2,.. ,..P' N ,..P'S}.

[0100] The answers to each question Q in the candidate preference data set generated by S1 are uneven, and all the answers in each list of candidate preference data D' are scored according to the reference document. More preferably, the score is a 1-10 point system, and the 1-10 point system includes five score ranges: 1-2 points: indicating that the answer is unavailable, 3-4 points: indicating that the answer meets the instruction requirements, 5-6 points: indicating that the answer is available, 7-8 points: indicating that the answer is good, and 9-10 points: indicating that the answer is perfect.

[0101] Score the answer by presetting a preference reply answer scoring prompt. The following is a detailed description in a specific scenario:

[0102] #Role Settings

[0103] Role X is a user answer quality analyst who determines the quality of answers based on user questions and reference document content.

[0104] #Task Description

[0105] We grade each answer based on the following criteria:

[0106] 1-2 points: The answer is basically unusable, the statement is unclear, there are serious factual errors, the instructions are not followed, and the answer is irrelevant;

[0107] 3-4 points: The answer meets the basic instruction requirements, but some minor errors occur and instruction requirements are omitted;

[0108] 5-6 points: The answer is basically acceptable, but the format and readability need to be improved, and there are some minor errors that are acceptable;

[0109] 7-8 points: This is basically a good answer, with no factual errors and no problem following instructions, but there is room for improvement in reasoning details and descriptive writing;

[0110] 9-10 points: This is basically a perfect answer. There is basically no room for improvement. At most, there are some harmless expressions or descriptions of preferences that vary from person to person.

[0111] Role X first needs to judge the quality of the answer based on the above standards, and carefully check the accuracy of the answer based on the question and reference documents.

[0112] Then the scoring basis and analysis are given, and finally the final score is given.

[0113] # Output requirements

[0114] Output the scoring basis and an answer score in strict accordance with the following JSON format:

[0115] {1

[0116] "score-reason": <Answer quality scoring basis and analysis>,

[0117] "score": <integer from 1 to 10>

[0118] }

[0119] # Input as follows:

[0120] ##User question: <User question content>

[0121] ##Reference document: <Reference document content>

[0122] ##Answers to be graded: <Answer content>.

[0123] More preferably, in S22, the k sampled answers are randomly shuffled, and each shuffled answer is assigned a preference identifier. Shuffling the order prevents the data pattern from being fixed, resulting in marking the first answer selected each time.

[0124] More preferably, in S22, each list candidate preference data D' samples the same number of answers in five score segments of 1 to 2, 3 to 4, ..., 9 to 10. This operation can ensure that the sampled answers are scored uniformly.

[0125] The S3 optimizes the loss function based on the output strategy of the restricted preference identification sorting data set, and then effectively and stably updates the SFT model parameters to obtain the preference alignment model.

[0126] Preferably, S3 includes:

[0127] S31, sort the data set according to the restricted preference identifiers, and obtain the output probability of each preference identifier in each list candidate preference data D' of each question Q , The calculation formula is:

[0128] ;

[0129] In the formula, q For a similar problem, y 1 ~ y k is the k answers sampled from the candidate preference data D' corresponding to the current similar question list, opt i is the preference mark of the corresponding answer;

[0130] S32. Output probability according to preference identification , calculate the loss between the preference pairs formed by any two preference identifiers in each list candidate preference data D' , add the losses between all preference pairs to get the loss of partial order prediction preference strategy optimization L rank , the calculation formula is as follows:

[0131] ;

[0132] is the Sigmoid function, exp is an exponential function, log is a logarithmic function;

[0133] S33. Calculate the probability score of the highest-scoring answer in each list of candidate preference data D' P(y c ｜q) , optimize the probability score of the SFT model in the current answer, and accumulate the probability scores of the highest-scoring answers in the S list candidate preference data D' to obtain the KL divergence , the calculation formula is as follows:

[0134] ;

[0135] ;

[0136] y c is the answer with the highest score in the list of candidate preference data D', Output for large models; Scores for the k answers sampled from the candidate preference data D' corresponding to the current similar question list, is the maximum value, P sft Based on P(y c ｜q) The probability score of the optimized SFT model in the corresponding answer;

[0137] S34. Optimize the loss of the partial order prediction preference strategy L rank and KL divergence Add together to get the final loss function L , the calculation formula is as follows:

[0138] ;

[0139] S35. Train the SFT model based on the final loss function to obtain a preference alignment model.

[0140] Finally, it should be noted that the above embodiments are only preferred embodiments of the present invention to illustrate the technical solutions of the present invention, rather than limiting them, and certainly not limiting the patent scope of the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention. In other words, any changes or modifications made to the main design concept and spirit of the present invention that have no substantive significance, and the technical problems they solve are still consistent with the present invention, should be included in the protection scope of the present invention. In addition, the direct or indirect application of the technical solutions of the present invention in other related technical fields is also included in the patent protection scope of the present invention.

Claims

1. A preference alignment model training method based on restricted preference identification sorting, characterized in that: The following steps are involved: S1. Construct candidate preference dataset based on SFT dataset; S2. Based on the candidate preference dataset, a restricted preference identification ranking dataset is constructed; S3, using the restricted preference identification sorting data set to train the SFT model and obtain the preference alignment model; S1 includes: S11, sampling N data pairs from the SFT data set, each data pair includes a question Q and an answer A; S12, based on the knowledge encyclopedia corpus text data set, generating a question set Q_{S} similar to each question Q by the SFT model, wherein the question set Q_{S} contains S similar sub-questions q, Q_{S}={q1, q2,.. ,..q N ,..q S }; S13. Retrieve a reference document associated with each question Q in the encyclopedia knowledge base. For each similar sub-question q of question Q, give m standard first answers, m second answers that are worse than the first answer, and m third answers that are better than the first answer according to the reference document; S14. Collect each similar sub-question q and the corresponding m first answers, m second answers, and m third answers into a list candidate preference data D'. Each question Q corresponds to S list candidate preference data D'. Collect the S list candidate preference data D' into a candidate preference data set D_{c}, D_{c}={D'1, D'2,.. ,..D' N ,..D' S }; The S2 includes: S21, based on the reference document, scoring all the first answers, second answers and third answers in each list candidate preference data D' of each question Q; S22, sampling k answers from each list candidate preference data D', k≤3m, assigning a preference identifier to each sampled answer, and outputting the preference identifier corresponding to the highest-scoring answer in each list candidate preference data D'; S23, repeating S22 for K times, and grouping the preference identifiers corresponding to the highest-scoring answers output each time into a restricted preference identifier ranking data P', and grouping the S restricted preference identifier ranking data P' of each question Q into a restricted preference identifier ranking data set P_{c}, P_{c}={P'1, P'2,.. ,..P' N ,..P' S }.

2. According to the method for training a preference alignment model based on restricted preference identification sorting according to claim 1, it is characterized in that: In S22, the k sampled answers are randomly shuffled, and then a preference identifier is assigned to each shuffled answer.

3. According to the method for training a preference alignment model based on restricted preference identification sorting according to claim 1, it is characterized in that: The scoring system is 1 to 10 points, which includes five score ranges: 1 to 2 points: indicating that the answer is not applicable, 3 to 4 points: indicating that the answer meets the instruction requirements, 5 to 6 points: indicating that the answer is applicable, 7 to 8 points: indicating that the answer is good, and 9 to 10 points: indicating that the answer is perfect.

4. According to the method for training a preference alignment model based on restricted preference identification sorting according to claim 3, it is characterized in that: In S22, each list of candidate preference data D' samples the same number of answers in five score segments of 1-2, 3-4, ..., 9-10.

5. According to the method for training a preference alignment model based on restricted preference identification sorting according to claim 1, it is characterized in that: The S3 includes: S31, sorting the data set according to restricted preference identifiers, and obtaining the output probability of each preference identifier in each list candidate preference data D' of each question Q; S32, calculating the loss between the preference pairs formed by any two preference identifiers in each list candidate preference data D' according to the output probability of the preference identifier, and adding the losses between all preference pairs to obtain the identifier partial order prediction preference strategy optimization loss; S33, calculating the probability score of the highest-scoring answer in each list candidate preference data D', optimizing the probability score of the SFT model in the current answer, and accumulating the probability scores of the highest-scoring answers in the S list candidate preference data D' to obtain the KL divergence; S34, adding the identifier partial order prediction preference strategy optimization loss and the KL divergence to obtain the final loss function; S35, training the SFT model based on the final loss function to obtain a preference alignment model.

Images