Question and answer method and device based on self-guided direct preference optimization, equipment and medium

Abstract

The application discloses a question and answer method and device based on self-guided direct preference optimization, equipment and medium, relates to the technical field of large language model, including: taking an initial large language model for completing a question and answer task as a to-be-updated strategy model, initializing a reference strategy model and a self-guided strategy model for regulating the gradient update direction of the to-be-updated strategy model; obtaining a subsequence of a positive example sequence and a subsequence of a negative example sequence in a historical question and answer scene; constructing a self-guided direct preference optimization loss function including a first contrast loss term between the positive example sequence and the negative example subsequence and a second contrast loss term between the positive example subsequence and the negative example sequence by using the above model; performing iterative gradient update on the to-be-updated strategy model by using the function to obtain a target large language model; inputting a current question and answer task into the target large language model to obtain a task processing result output by the target large language model. The ability of the model to generate human preference responses is improved, and the training stability is enhanced.

Description

Technical Field

[0001] This invention relates to the field of large language model technology, and in particular to a question-answering method, apparatus, device and medium based on self-guided direct preference optimization. Background Technology

[0002] Large Language Models (LLMs) can be used to complete question-answering tasks. For example, if it is a large financial dialogue model trained, the model can be deployed in financial dialogue scenarios. The trained model can generate answers that conform to human preferences based on the user's historical investment preferences (conservative / aggressive). If it is a large financial question recommendation model, the model can be deployed in financial recommendation application scenarios. Based on user profiles (risk preferences, investment horizon, return goals, etc.), it can generate personalized financial product recommendation lists or investment portfolio suggestions.

[0003] In large language model training, existing techniques for aligning model output with human preferences primarily rely on Reinforcement Learning from Human Feedback (RLHF). This method comprises three main steps: Supervised Fine-Tuning (SFT), reward learning, and Reinforcement Learning (RL). However, traditional RLHF suffers from a complex training process due to its heavy dependence on the reward model for reinforcement learning optimization. Training a high-quality reward model requires significant human and time investment, and consideration must be given to data quality and diversity. Furthermore, the reward model is susceptible to data bias and noise, resulting in unstable performance.

[0004] To simplify the training process, Direct Preference Optimization (DPO) was proposed. It reparameterizes the reward model, mapping the reward function to the optimal policy, thus eliminating the reward training step and transforming the online reward model's preference feedback into offline implicit modeling. However, DPO suffers from high computational costs, generates verbose text, is prone to overfitting, has limited ability to generate responses that conform to human preferences, and its effectiveness is unstable, making it difficult to fully meet the needs of practical applications.

[0005] In summary, improving the model's ability to generate human-preferred responses and enhancing training stability are problems that need to be solved in this field. Summary of the Invention

[0006] In view of this, the purpose of this invention is to provide a question-answering method, apparatus, device, and medium based on self-guided direct preference optimization, which improves the model's ability to generate human preference responses and enhances training stability. The specific solution is as follows:

[0007] Firstly, this application discloses a question-answering method based on self-guided direct preference optimization, including:

[0008] The initial large language model used to complete the question-answering task is determined as the policy model to be updated, and a reference policy model and a self-guided policy model are initialized; wherein, the self-guided policy model is used to regulate the gradient update direction of the policy model to be updated.

[0009] Obtain positive example sequences of positive example sequences and negative example sequences of negative example sequences from training data in historical question-answering scenarios;

[0010] A self-guided direct preference optimization loss function is constructed using the policy model to be updated, the reference policy model, and the self-guided policy model. This loss function includes a first contrastive loss term between the positive example sequence and the negative example sequence and a second contrastive loss term between the positive example sequence and the negative example sequence. The first contrastive loss term and the second contrastive loss term are symmetric to each other.

[0011] The parameters of the policy model to be updated are iteratively updated using the self-guided direct preference optimization loss function to determine the target policy model as the target large language model.

[0012] Input the current question-answering task into the target large language model to obtain the task processing result output by the target large language model.

[0013] Optionally, obtaining the positive example sequence of the positive example sequence and the negative example sequence of the negative example sequence from the training data in the historical question-answering scenario includes:

[0014] Obtain training data from historical question-and-answer scenarios; wherein, the training data includes positive and negative examples as model outputs;

[0015] The positive example data and the negative example data are respectively lexicalized to obtain positive example sequences and negative example sequences containing each lexical unit;

[0016] Extract positive example sequences and negative example sequences from the positive example sequences and the negative example sequences, respectively.

[0017] Accordingly, the iterative gradient update of the parameters of the policy model to be updated using the self-guided direct preference optimization loss function includes:

[0018] The parameters of the policy model to be updated are updated using the self-guided direct preference optimization loss function to obtain a new policy model to be updated. If the preset stopping iteration condition is not met, the process jumps back to the step of extracting positive and negative example sequences from the positive and negative example sequences, respectively.

[0019] Optionally, the step of extracting positive example sequences and negative example sequences from the positive example sequences and the negative example sequences, respectively, includes:

[0020] The first truncation length of the positive example sequence and the second truncation length of the negative example sequence are determined based on the sequence lengths of the positive example sequence and the negative example sequence and the preset subsequence sampling ratio;

[0021] A continuous segment extracted from the positive example sequence based on the first truncation length is determined as a positive example sequence, and a continuous segment extracted from the negative example sequence based on the second truncation length is determined as a negative example sequence.

[0022] Optionally, the step of using the self-guided direct preference optimization loss function to perform gradient updates on the parameters of the policy model to be updated, in order to obtain a new policy model to be updated, includes:

[0023] Based on the parameters of the policy model to be updated, the derivative of the self-guided direct preference optimization loss function is taken to obtain the gradient characterizing the direction and magnitude of the parameter update.

[0024] By backpropagation and updating the parameters of the policy model to be updated along the opposite direction of the gradient, the self-guided direct preference optimization loss function is minimized to obtain a new policy model to be updated.

[0025] Optionally, the reference policy model is used to constrain the log odds of the policy model to be updated;

[0026] Accordingly, the first contrastive loss term is constructed, including:

[0027] The first complete log-probability of the positive example sequence is constructed using the policy model to be updated and the reference policy model, and the first subsequence log-probability of the negative example sequence is constructed using the self-guided policy model and the reference policy model.

[0028] A first contrastive loss term is constructed using the difference between the first complete log-log odds and the first subsequence log-log odds.

[0029] Optionally, a second contrastive loss term is constructed, including:

[0030] The second subsequence log odds of the positive example sequence are constructed using the self-guided strategy model and the reference strategy model, and the second complete log odds of the negative example sequence are constructed using the strategy model to be updated and the reference strategy model.

[0031] A second contrastive loss term is constructed using the difference between the log-odds of the second subsequence and the log-odds of the second complete sequence.

[0032] Optionally, the self-guided direct preference optimization loss function is:

[0033] ;

[0034] in, The first complete logarithmic probability, The log-probability of the first subsequence. The log-probability of the second subsequence. The second complete logarithmic probability, For the Sigmoid function, For hyperparameters, For expectations;

[0035] The formulas for obtaining the first complete logarithmic odds, the first subsequence logarithmic odds, the second subsequence logarithmic odds, and the second complete logarithmic odds are as follows:

[0036] , , and ;

[0037] in, For the strategy model to be updated, For reference strategy model, This is a self-guided strategy model. This is a positive example sequence. A sequence of positive examples. This is a counterexample sequence. For counterexample sequences, This refers to the cue data used as input to the model in the training data.

[0038] Secondly, this application discloses a question-answering device based on self-guided direct preference optimization, comprising:

[0039] The strategy model initialization module is used to determine the initial large language model used to complete the question answering task as the strategy model to be updated, and to initialize the reference strategy model and the self-guided strategy model; wherein, the self-guided strategy model is used to regulate the gradient update direction of the strategy model to be updated.

[0040] The subsequence sampling module is used to obtain positive example sequences of positive example sequences and negative example sequences of negative example sequences from the training data of historical question-answering scenarios.

[0041] The loss function construction module is used to construct a self-guided direct preference optimization loss function, which includes a first contrastive loss term between the positive example sequence and the negative example sequence and a second contrastive loss term between the positive example sequence and the negative example sequence, using the policy model to be updated, the reference policy model, and the self-guided policy model; wherein the first contrastive loss term and the second contrastive loss term are symmetrical to each other;

[0042] The gradient update module is used to iteratively update the parameters of the policy model to be updated using the self-guided direct preference optimization loss function, so as to determine the target policy model as the target large language model.

[0043] The question-answering task processing module is used to input the current question-answering task into the target large language model to obtain the task processing result output by the target large language model.

[0044] Thirdly, this application discloses an electronic device, including:

[0045] Memory, used to store computer programs;

[0046] A processor is configured to execute the computer program to implement the steps of the aforementioned disclosed question-answering method based on self-guided direct preference optimization.

[0047] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the aforementioned disclosed question-answering method based on self-guided direct preference optimization.

[0048] The beneficial effects of this application are as follows: This application determines the initial large language model used to complete the question-answering task as the policy model to be updated, and initializes a reference policy model and a self-guided policy model; wherein, the self-guided policy model is used to regulate the gradient update direction of the policy model to be updated; positive example sequences of positive example sequences and negative example sequences of negative example sequences are obtained from the training data in historical question-answering scenarios; a self-guided direct preference optimization loss function is constructed using the policy model to be updated, the reference policy model, and the self-guided policy model, including a first contrastive loss term between the positive example sequence and the negative example sequence and a second contrastive loss term between the positive example sequence and the negative example sequence; wherein, the first contrastive loss term and the second contrastive loss term are symmetrical structures; the parameters of the policy model to be updated are iteratively updated using the self-guided direct preference optimization loss function to determine the target policy model as the target large language model; the current question-answering task is input into the target large language model to obtain the task processing result output by the target large language model. Therefore, this application achieves precise control over the model training process by using the initial large language model as the policy model to be updated and combining it with a reference policy model and a self-guided policy model. The self-guided policy model effectively balances the optimization process of positive and negative example sequences by adjusting the gradient update direction. By extracting positive and negative example sequences from historical question-and-answer data, a symmetrical first and second contrastive loss term is constructed, enabling the model to simultaneously consider the preference alignment of positive and negative examples during training. This symmetrical loss design avoids the problem of excessive decrease in negative example rewards in traditional methods, improving the stability and efficiency of training. Iterative gradient updates are performed using the self-guided direct preference optimization loss function, and the final target large language model can more accurately handle the current question-and-answer task, generate high-quality answers that conform to user preferences, simplify the training process, reduce the dependence on the reward model, and significantly improve the model's alignment performance and generation effect through subsequence sampling and symmetrical loss design. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0050] Figure 1 This is a flowchart of a question-answering method based on self-guided direct preference optimization disclosed in this application;

[0051] Figure 2This application discloses a specific DPO gradient flow graph for selected and rejected rewards;

[0052] Figure 3 This is a schematic diagram of a question-answering device based on self-guided direct preference optimization disclosed in this application;

[0053] Figure 4 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0054] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0055] Large language models can be used to complete question-answering tasks. For example, if it is a large financial dialogue model trained, the model can be deployed in a financial dialogue scenario. The trained model can generate answers that conform to human preferences based on the user's historical investment preferences (conservative / aggressive). If it is a large financial question recommendation model, the model can be deployed in a financial recommendation application scenario. Based on the user profile (risk preference, investment period, return target, etc.), it can generate a personalized list of financial product recommendations or investment portfolio suggestions.

[0056] In large language model training, existing techniques for aligning model output with human preferences primarily rely on reinforcement learning based on human feedback. This approach includes three main steps: supervised fine-tuning, reward learning, and reinforcement learning. However, traditional RLHF suffers from a complex training process due to its heavy dependence on reward models for optimization. Training high-quality reward models requires significant human and time investment, and consideration must be given to data quality and diversity. Furthermore, reward models are susceptible to data bias and noise, resulting in unstable performance.

[0057] To simplify the training process, Direct Preference Optimization (DPO) was proposed. It reparameterizes the reward model, mapping the reward function to the optimal policy, thus eliminating the reward training step and transforming the online reward model's preference feedback into offline implicit modeling. However, DPO suffers from high computational costs, verbose generated text, overfitting, limited ability to generate responses that conform to human preferences, and unstable effectiveness, making it difficult to fully meet practical application needs.

[0058] To address this, this application provides a question-answering scheme based on self-guided direct preference optimization, which improves the model's ability to generate human preference responses and enhances training stability.

[0059] See Figure 1As shown in the figure, this application discloses a question-answering method based on self-guided direct preference optimization, including:

[0060] Step S11: Determine the initial large language model used to complete the question-answering task as the policy model to be updated, and initialize the reference policy model and the self-guided policy model; wherein, the self-guided policy model is used to regulate the gradient update direction of the policy model to be updated.

[0061] An initial large language model is constructed to complete the question-answering task. Specifically, this initial large language model can be a large financial dialogue model, which is then deployed in a financial dialogue scenario. The trained model can generate answers that conform to human preferences based on the user's historical investment preferences (conservative / aggressive), improving the accuracy and professionalism of customer service responses, reducing the pressure on human customer service, and further ensuring that answers comply with financial regulatory requirements through preference alignment optimization. Alternatively, it can be a large financial recommendation model, which is then deployed in a financial recommendation application scenario. Based on user profiles (risk preference, investment period, return target, etc.), it generates personalized financial product recommendation lists or investment portfolio suggestions. Through gradient update optimization, the length bias of recommendation results is reduced, information transmission efficiency is improved, the conversion rate and user satisfaction of the recommendation system are increased, and the risk of customer churn caused by "false recommendations" is reduced.

[0062] The initial large language model used to complete the question-answering task is identified as the policy model to be updated. and initialize the reference policy model. and self-guided strategy model The self-guided policy model is used to regulate the gradient update direction of the policy model to be updated, while the reference policy model is used to constrain the log odds of the policy model to be updated, that is, to constrain the KL divergence.

[0063] Step S12: Obtain positive example sequences of positive example sequences and negative example sequences of negative example sequences from the training data in historical question-answering scenarios.

[0064] In this embodiment, obtaining positive example sequences of positive example sequences and negative example sequences of negative example sequences from training data in historical question-answering scenarios includes: obtaining training data in historical question-answering scenarios; wherein, the training data includes positive example data and negative example data as model output; performing lexicalization on the positive example data and the negative example data respectively to obtain positive example sequences and negative example sequences containing each lexical; and extracting positive example sequences and negative example sequences from the positive example sequences and the negative example sequences respectively.

[0065] Obtain training data in the historical Q&A scenario, where the training data specifically includes data as the model input and data as the model output. The data as the model input is the historical Q&A task x (i.e., the input text sequence), such as "User A has some idle funds. How should they be allocated?", "What financial products are suitable for conservative investors?", etc. The data as the model output includes positive example data y that conforms to human preferences w and negative example data y that conforms to human rejection l , that is, the positive example data is the correct sample that conforms to the expected Q&A task, while the negative example data is the opposite. However, the negative example data can help the model avoid wrong directions. For example, the Q&A task is "User A has some idle funds. How should they be allocated?", and its positive example data is "①Allocate 30% to low-risk money funds to ensure liquidity; ②Allocate 40% to balanced hybrid funds for long-term appreciation; ③Allocate 30% to high-growth industries, and pay attention to adjusting the proportion according to personal risk tolerance", and its negative example data is "Buy all high-risk and high-appreciation funds, and it will triple next year", etc. So a set of training data is [x, y w , y l . Tokenize the positive example data and the negative example data respectively to obtain a positive example sequence and a negative example sequence containing each token. For example, if the positive example data is "Should leave early today", the positive example sequence after tokenization is ["Today", "Should", "Leave", "Early"]; from the positive example sequence y w , the negative example sequence y l respectively intercept the positive example subsequence , the negative example subsequence .

[0066] In this embodiment, the step of respectively intercepting the positive example subsequence and the negative example subsequence from the positive example sequence and the negative example sequence includes: determining the first interception length of the positive example sequence and the second interception length of the negative example sequence based on the sequence lengths of the positive example sequence and the negative example sequence and a preset subsequence sampling ratio; determining the continuous segment intercepted from the positive example sequence based on the first interception length as the positive example subsequence, and determining the continuous segment intercepted from the negative example sequence based on the second interception length as the negative example subsequence.

[0067] To simplify the calculation, introduce and . Here x is the input text sequence, and y is the complete output text sequence. Let the token sequence length of y be T, y t represents the token at the t-th index, and y<t represents all tokens before the t-th index. Since is a subsequence of y, and considering that can be represented at the token level as , where * belongs to the set , can be represented as X2=p2Y2 and X1=p1Y1, where p1 and p2 represent the product of the label probability ratios of the remaining labels in sequences X1 and X2 excluding subsequences Y1 and Y2.

[0068] The sequence length of the positive example sequence is l1, and the sequence length of the negative example sequence is l2. c The preset subsequence sampling ratio can be set separately for positive and negative sequences, where l1 is the smaller of l2 and l2 is the smaller value. For example, if the preset subsequence sampling ratio for the positive sequence is r1 and the preset subsequence sampling ratio for the negative sequence is r2, then l c ×r1 is the first truncation length of the positive example sequence, l c ×r2 is the second truncation length of the negative example sequence. Then, the continuous segments truncated from the positive example sequence based on the first truncation length are determined as the positive example sequence. And the continuous segments extracted from the counterexample sequence based on the second truncation length are determined as the counterexample sequence. .

[0069] Step S13: Construct a self-guided direct preference optimization loss function using the policy model to be updated, the reference policy model, and the self-guided policy model, including a first contrastive loss term between the positive example sequence and the negative example sequence and a second contrastive loss term between the positive example sequence and the negative example sequence; wherein the first contrastive loss term and the second contrastive loss term are symmetrical structures.

[0070] Traditional Direct Policy Allocation (DPO) is a widely adopted technique for optimizing LLM preferences. This method stands out because it innovatively utilizes an analytical mapping to transform the reward function into the optimal policy, simplifying the alignment process without directly using the reward model. The cornerstone of DPO lies in its specific transformation, the mathematical formula of which is as follows:

[0071] ;Formula (1)

[0072] Among them, is Reward function, It is a scaling factor. This represents the strategy inferred from the reward model. This represents the reference strategy. Here, Z(x) is a normalization constant to ensure that the probabilities are scaled appropriately. The DPO will prompt for instructions given the input. In the case of selecting the result Compared to rejection results The probability of being more preferred is expressed as follows:

[0073] ;Formula (2)

[0074] Formula (2) quantifies the relative preference between two responses by comparing their relevant reward values. Within this probabilistic framework, the loss function of DPO (denoted as...) )as follows:

[0075] ;Formula (3)

[0076] and The definition is as follows:

[0077] ;Formula (4)

[0078] in, This represents the sigmoid function. As a scaling factor, equations (3) and (4) encapsulate the principles of the Bradley-Terry model, integrating preference data into the learning process. In this way, DPO ensures that the responses of LLMs are consistent with observed human preferences.

[0079] Given a selected reward and refusing rewards ,about and The partial derivatives are as follows:

[0080] ;Formula (5)

[0081] ;Formula (6)

[0082] Furthermore, the formula for the ratio of the increase in the probability of a human's preferred response to the decrease in the probability of a human's dislike response is as follows:

[0083] ;Formula (7)

[0084] The chosen reward increases the log probability of the model outputting a human-preferred response. This is reflected in rewarding preferred responses during training, increasing the model's probability of producing such responses. Conversely, the rejected reward decreases the log probability of an undesirable response. This is reflected in penalizing undesirable responses during training, thus reducing the model's probability of producing such responses.

[0085] For example Figure 2 The diagram shows a gradient flow graph of a specific DPO for selected and rejected rewards. In a traditional DPO, when When the value is small, the DPO gradient flow tends to decrease rapidly. , and for This behavior, involving only minor adjustments, limits the ability of LLMs to effectively generate highly preferred responses. As DPO optimization progresses, the selected reward... Increase, but refuse rewards Reduce, therefore, / <1. According to formula (7), this results in gradient updates for rejected rewards being faster than gradient updates for selected rewards. The above theoretical framework points to several directions for improving DPO:

[0086] G1: Preventing Rejection of Rewards It should drop rapidly to a very small value, or it may stop responding to the selected reward. A meaningful update.

[0087] G2: Increase the ratio in formula (7) to make a more substantial update to the selected reward, thereby enhancing the ability of LLMs to generate preference responses.

[0088] These adjustments aim to optimize the optimization process of RLHF and improve the performance of LLM in aligning with human preferences. To achieve these goals, it is proposed to incorporate an adjusted preference optimization objective, namely the self-guided policy model, into the original SGDPO loss function.

[0089] In this embodiment, constructing the first contrastive loss term includes: constructing the first complete log-odds of the positive example sequence using the policy model to be updated and the reference policy model, and constructing the first subsequence log-odds of the negative example sequence using the self-guided policy model and the reference policy model; and constructing the first contrastive loss term using the difference between the first complete log-odds and the first subsequence log-odds.

[0090] Utilizing the policy model to be updated Reference strategy model Constructing a sequence of positive examples The first complete log odds, and utilize a self-guided strategy model. Reference strategy model Constructing a sequence of counterexamples The first subsequence log odds; the first contrastive loss term is constructed using the difference between the first complete log odds and the first subsequence log odds, which encourages higher rewards for positive examples than for negative examples in the subsequence.

[0091] In this embodiment, constructing the second contrastive loss term includes: constructing the second subsequence log odds of the positive example sequence using the self-guided policy model and the reference policy model, and constructing the second complete log odds of the negative example sequence using the policy model to be updated and the reference policy model; and constructing the second contrastive loss term using the difference between the second subsequence log odds and the second complete log odds.

[0092] Using self-guided strategy model Reference strategy model Constructing a sequence of positive examples The second subsequence log odds, and the policy model to be updated. Reference strategy model Constructing a sequence of counterexamples The second complete log odds; a second contrastive loss term is constructed using the difference between the second subsequence log odds and the second complete log odds, which encourages positive subsequences to outnumber negative complete subsequences.

[0093] In this embodiment, the self-guided direct preference optimization loss function is:

[0094] ;Formula (8)

[0095] in, The first complete logarithmic probability, The log-probability of the first subsequence. The log-probability of the second subsequence. The second complete logarithmic probability, For the Sigmoid function, For hyperparameters, For expectations;

[0096] In this embodiment, the formulas for obtaining the first complete logarithmic odds, the first subsequence logarithmic odds, the second subsequence logarithmic odds, and the second complete logarithmic odds are as follows:

[0097] , , and ;

[0098] in, For the strategy model to be updated, For reference strategy model, This is a self-guided strategy model. This is a positive example sequence. A sequence of positive examples. This is a counterexample sequence. For counterexample sequences, This refers to the cue data used as input to the model in the training data.

[0099] First, the initial form of the self-guided direct preference optimization loss function is as follows:

[0100] ;Formula (9)

[0101] in, The definition is as follows:

[0102] ;Formula (10)

[0103] To simplify the calculation, log odds are introduced:

[0104] , , , .

[0105] Step S14: Use the self-guided direct preference optimization loss function to iteratively update the parameters of the policy model to be updated, so as to determine the obtained target policy model as the target large language model.

[0106] In this embodiment, the step of iteratively updating the parameters of the policy model to be updated using the self-guided direct preference optimization loss function includes: updating the parameters of the policy model to be updated using the self-guided direct preference optimization loss function to obtain a new policy model to be updated; if the preset stopping iteration condition is not met, the process jumps back to the step of extracting positive and negative example sequences from the positive and negative example sequences, respectively.

[0107] It is understandable that the parameters of the policy model to be updated are updated using the constructed self-guided direct preference optimization loss function to obtain a new policy model to be updated. If the current time does not meet the preset stopping iteration condition, the process jumps back to the step of extracting positive and negative example sequences from the positive and negative example sequences respectively. Specifically, the preset stopping iteration condition can be that the current iteration number reaches the preset iteration threshold, or the convergence of the new policy model to be updated meets the preset convergence condition.

[0108] In this embodiment, the step of using the self-guided direct preference optimization loss function to perform gradient updates on the parameters of the policy model to be updated to obtain a new policy model to be updated includes: taking the derivative of the self-guided direct preference optimization loss function with respect to the parameters of the policy model to be updated to obtain a gradient representing the direction and magnitude of the parameter update; updating the parameters of the policy model to be updated in the opposite direction of the gradient through backpropagation to minimize the self-guided direct preference optimization loss function and obtain a new policy model to be updated.

[0109] The parameters of the policy model to be updated are differentiated with respect to the self-guided direct preference optimization loss function to obtain the gradient representing the direction and magnitude of parameter updates. Through backpropagation, the parameters of the policy model to be updated are updated in the opposite direction of the gradient to minimize the self-guided direct preference optimization loss function and obtain a new policy model to be updated. Specifically, backpropagation can be performed using automatic differentiation.

[0110] Step S15: Input the current question-answering task into the target large language model to obtain the task processing result output by the target large language model.

[0111] The final large language model is the target large language model. A guiding term is introduced into the traditional loss function to guide the training gradient flow during the optimization process and control the updates of the selected reward and the rejection reward. A loss function (i.e., a self-guided direct preference optimization loss function) is constructed based on the objective function. The loss function contains computational terms related to the guiding policy model and is used to adjust the model's parameter updates. Subsequences are constructed from the original sequence, and the gradient changes of the selected reward and the rejection reward are affected by adjusting the hyperparameters related to the length of the subsequences, thereby optimizing the model training.

[0112] Next, the target large language model can be used to process the current question-answering task, thereby obtaining the task processing result output by the large model, and the task processing result conforms to human preferences.

[0113] The beneficial effects of this application are as follows: This application determines the initial large language model used to complete the question-answering task as the policy model to be updated, and initializes a reference policy model and a self-guided policy model; wherein, the self-guided policy model is used to regulate the gradient update direction of the policy model to be updated; positive example sequences of positive example sequences and negative example sequences of negative example sequences are obtained from the training data in historical question-answering scenarios; a self-guided direct preference optimization loss function is constructed using the policy model to be updated, the reference policy model, and the self-guided policy model, including a first contrastive loss term between the positive example sequence and the negative example sequence and a second contrastive loss term between the positive example sequence and the negative example sequence; wherein, the first contrastive loss term and the second contrastive loss term are symmetrical structures; the parameters of the policy model to be updated are iteratively updated using the self-guided direct preference optimization loss function to determine the target policy model as the target large language model; the current question-answering task is input into the target large language model to obtain the task processing result output by the target large language model. Therefore, this application achieves precise control over the model training process by using the initial large language model as the policy model to be updated and combining it with a reference policy model and a self-guided policy model. The self-guided policy model effectively balances the optimization process of positive and negative example sequences by adjusting the gradient update direction. By extracting positive and negative example sequences from historical question-and-answer data, a symmetrical first and second contrastive loss term is constructed, enabling the model to simultaneously consider the preference alignment of positive and negative examples during training. This symmetrical loss design avoids the problem of excessive decrease in negative example rewards in traditional methods, improving the stability and efficiency of training. Iterative gradient updates are performed using the self-guided direct preference optimization loss function, and the final target large language model can more accurately handle the current question-and-answer task, generate high-quality answers that conform to user preferences, simplify the training process, reduce the dependence on the reward model, and significantly improve the model's alignment performance and generation effect through subsequence sampling and symmetrical loss design.

[0114] The self-guided direct preference optimization loss function in this embodiment has significant advantages in enhancing preference optimization. The following is a demonstration of the advantages of the self-guided direct preference optimization loss function:

[0115] first, about and The partial derivatives are as follows:

[0116] ;Formula (11)

[0117] As can be seen from formula (11), and The gradients depend on Y1 and Y2, respectively. Therefore, by manipulating Y1 and Y2, the gradient flow in the alignment method can be controlled, thereby affecting the updates to the selected and rejected rewards.

[0118] In addition, for The following formula is given:

[0119] ;Formula (12)

[0120] In addition, for The following formula is given:

[0121] ;Formula (13)

[0122] Secondly, partial derivatives The partial derivative increases with the increase of Y2. As Y1 increases, it decreases; during fine-tuning, the preference alignment algorithm increases the probability of generating preferred text while decreasing the probability of generating unpreferred text, thus p2 < 1. Therefore, Y2 > X2. Comparing Equation 11 with Equations 5 and 6, we can see the difference lies in only one variable. For example, X2 in Equation 5 is replaced by Y2 in Equation 11. Therefore, we can obtain the following formula:

[0123] ;Formula (14)

[0124] Equation (14) shows that SGDPO can mitigate the rapid decline in the rejection reward. Clearly, as the model converges, both the selected and rejected rewards maintain higher values ​​compared to the DPO baseline. Therefore, SGDPO enhances the updating of the selected reward, thereby facilitating the generation of the preference response.

[0125] Among them, for The following formula is given:

[0126] ;Formula (15)

[0127] In addition, for The following formula is given:

[0128] ;Formula (16)

[0129] Next, set = Let z = Y1 / Y2. For each pairwise preference instance, the ratio of the increase in the probability of a human preference response to the decrease in the probability of a human non-preference response is:

[0130] ;Formula (17)

[0131] Among them, for It has the following definition:

[0132] ;Formula (18)

[0133] f(z) is a monotonically increasing function of z. When p1p2 < 1, f(z) increases with increasing z; conversely, when p1p2 > 1, f(z) decreases with increasing z. Furthermore, when p1p2 < 1, f(z) > 1. If the rejection reward decreases rapidly, it will cause p1p2 < 1. This will make f(z) > 1, thereby increasing the ratio given by formula (17). In addition, during training, as z = y1 / y2 increases, f(z) also increases. This behavior is consistent with the objective G2, thereby enhancing the ability of LLM to generate preferred text.

[0134] Regarding the above content, the following formulas exist:

[0135] ;Formula (19)

[0136] Let X2 = p2Y2 and X1 = p1Y1, z = y1 / y2, then we have the following formula:

[0137] ;Formula (20)

[0138] Furthermore: the derivative of f(z) with respect to z is:

[0139] ;Formula (21)

[0140] Since z = y1 / y2, does it exist? , The value of f(z) depends on the values ​​of p1 and p2. Therefore, when p1p2 < 1, the function f(z) is monotonically increasing, and conversely, when p1p2 > 1, the function f(z) is monotonically decreasing.

[0141] In summary, from sequence y respectively w and y l Derived subsequence and . and As indicators, they are used to guide the refinement of updates for selected and rejected reward adjustments. Let l1 and l2 be the sequences y... w and y l The length of l c Defined as the smaller of l1 and l2. From (y w y l In the pair, randomly select a length of (l) c ×r1,l c ×r2) preference data pairs ( , ̂), where r1 and r2 are hyperparameters. Adjusting r1 and r2 can affect the gradient changes associated with the selected reward and the rejected reward during preference optimization. For example, decreasing r2 will cause r1 to increase, since it is derived from samples of rejected rewards, which will further enhance the update magnitude for the selected reward.

[0142] See Figure 3 As shown in the figure, this application discloses a question-answering device based on self-guided direct preference optimization, comprising:

[0143] The strategy model initialization module 11 is used to determine the initial large language model used to complete the question answering task as the strategy model to be updated, and to initialize the reference strategy model and the self-guided strategy model; wherein, the self-guided strategy model is used to regulate the gradient update direction of the strategy model to be updated.

[0144] The subsequence sampling module 12 is used to obtain positive example sequences of positive example sequences and negative example sequences of negative example sequences from the training data in historical question-answering scenarios;

[0145] Loss function construction module 13 is used to construct a self-guided direct preference optimization loss function, including a first contrastive loss term between the positive example sequence and the negative example sequence and a second contrastive loss term between the positive example sequence and the negative example sequence, using the policy model to be updated, the reference policy model, and the self-guided policy model; wherein the first contrastive loss term and the second contrastive loss term are symmetrical to each other;

[0146] The gradient update module 14 is used to iteratively update the parameters of the policy model to be updated using the self-guided direct preference optimization loss function, so as to determine the obtained target policy model as the target large language model.

[0147] The question-answering task processing module 15 is used to input the current question-answering task into the target large language model to obtain the task processing result output by the target large language model.

[0148] The beneficial effects of this application are as follows: This application determines the initial large language model used to complete the question-answering task as the policy model to be updated, and initializes a reference policy model and a self-guided policy model; wherein, the self-guided policy model is used to regulate the gradient update direction of the policy model to be updated; positive example sequences of positive example sequences and negative example sequences of negative example sequences are obtained from the training data in historical question-answering scenarios; a self-guided direct preference optimization loss function is constructed using the policy model to be updated, the reference policy model, and the self-guided policy model, including a first contrastive loss term between the positive example sequence and the negative example sequence and a second contrastive loss term between the positive example sequence and the negative example sequence; wherein, the first contrastive loss term and the second contrastive loss term are symmetrical structures; the parameters of the policy model to be updated are iteratively updated using the self-guided direct preference optimization loss function to determine the target policy model as the target large language model; the current question-answering task is input into the target large language model to obtain the task processing result output by the target large language model. Therefore, this application achieves precise control over the model training process by using the initial large language model as the policy model to be updated and combining it with a reference policy model and a self-guided policy model. The self-guided policy model effectively balances the optimization process of positive and negative example sequences by adjusting the gradient update direction. By extracting positive and negative example sequences from historical question-and-answer data, a symmetrical first and second contrastive loss term is constructed, enabling the model to simultaneously consider the preference alignment of positive and negative examples during training. This symmetrical loss design avoids the problem of excessive decrease in negative example rewards in traditional methods, improving the stability and efficiency of training. Iterative gradient updates are performed using the self-guided direct preference optimization loss function, and the final target large language model can more accurately handle the current question-and-answer task, generate high-quality answers that conform to user preferences, simplify the training process, reduce the dependence on the reward model, and significantly improve the model's alignment performance and generation effect through subsequence sampling and symmetrical loss design.

[0149] Furthermore, embodiments of this application also provide an electronic device. Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0150] Figure 4This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Specifically, it may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the self-guided direct preference optimization-based question-answering method disclosed in any of the foregoing embodiments.

[0151] In this embodiment, the power supply 23 is used to provide operating voltage for various hardware devices on the electronic device; the communication interface 24 can create a data transmission channel between the electronic device and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0152] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0153] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored on it include operating system 221, computer program 222 and data 223, etc., and the storage method can be temporary storage or permanent storage.

[0154] The operating system 221 manages and controls the various hardware devices and computer programs 222 on the electronic device to enable the processor 21 to perform calculations and processing on the massive amounts of data 223 in the memory 22. The operating system can be Windows, Unix, Linux, etc. The computer program 222, in addition to including a computer program capable of performing the self-booted direct preference optimization-based question-and-answer method executed by the electronic device as disclosed in any of the foregoing embodiments, may further include computer programs capable of performing other specific tasks. The data 223 may include data received by the electronic device from external devices, as well as data collected by its own input / output interface 25.

[0155] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned question-answering method based on self-guided direct preference optimization. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0156] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0157] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly in hardware, software modules executed by a processor, or a combination of both. The software module may be located in random access memory (RAM), memory, read-only memory (ROM), electrically programmable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), register, hard disk, removable disk, CD-ROM (Compact Disc Read-Only Memory), or any other form of storage medium known in the art.

[0158] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0159] The above provides a detailed description of a question-answering method, apparatus, device, and medium based on self-guided direct preference optimization provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A question-answering method based on self-guided direct preference optimization, characterized in that, include: The initial large language model used to complete the question-answering task is determined as the policy model to be updated, and a reference policy model and a self-guided policy model are initialized; wherein, the reference policy model is used to constrain the log odds of the policy model to be updated, and the self-guided policy model is used to regulate the gradient update direction of the policy model to be updated. Obtain positive example sequences of positive example sequences and negative example sequences of negative example sequences from training data in historical question-answering scenarios; A self-guided direct preference optimization loss function is constructed using the policy model to be updated, the reference policy model, and the self-guided policy model. This loss function includes a first contrastive loss term between the positive example sequence and the negative example sequence and a second contrastive loss term between the positive example sequence and the negative example sequence. The first contrastive loss term and the second contrastive loss term are symmetric to each other. The parameters of the policy model to be updated are iteratively updated using the self-guided direct preference optimization loss function to determine the target policy model as the target large language model. Input the current question-answering task into the target large language model to obtain the task processing result output by the target large language model.

2. The question-answering method based on self-guided direct preference optimization according to claim 1, characterized in that, The step of obtaining positive example sequences and negative example sequences of positive example sequences from training data in historical question-answering scenarios includes: Obtain training data from historical question-and-answer scenarios; wherein, the training data includes positive and negative examples as model outputs; The positive example data and the negative example data are respectively lexicalized to obtain positive example sequences and negative example sequences containing each lexical unit; Extract positive example sequences and negative example sequences from the positive example sequences and the negative example sequences, respectively. Accordingly, the iterative gradient update of the parameters of the policy model to be updated using the self-guided direct preference optimization loss function includes: The parameters of the policy model to be updated are updated using the self-guided direct preference optimization loss function to obtain a new policy model to be updated. If the preset stopping iteration condition is not met, the process jumps back to the step of extracting positive and negative example sequences from the positive and negative example sequences, respectively.

3. The question-answering method based on self-guided direct preference optimization according to claim 2, characterized in that, The step of extracting positive example sequences and negative example sequences from the positive example sequences and the negative example sequences, respectively, includes: The first truncation length of the positive example sequence and the second truncation length of the negative example sequence are determined based on the sequence lengths of the positive example sequence and the negative example sequence and the preset subsequence sampling ratio; A continuous segment extracted from the positive example sequence based on the first truncation length is determined as a positive example sequence, and a continuous segment extracted from the negative example sequence based on the second truncation length is determined as a negative example sequence.

4. The question-answering method based on self-guided direct preference optimization according to claim 2, characterized in that, The step of using the self-guided direct preference optimization loss function to perform gradient updates on the parameters of the policy model to be updated, in order to obtain a new policy model to be updated, includes: Based on the parameters of the policy model to be updated, the derivative of the self-guided direct preference optimization loss function is taken to obtain the gradient characterizing the direction and magnitude of the parameter update. By backpropagation and updating the parameters of the policy model to be updated along the opposite direction of the gradient, the self-guided direct preference optimization loss function is minimized to obtain a new policy model to be updated.

5. The question-answering method based on self-guided direct preference optimization according to any one of claims 1 to 4, characterized in that, Construct the first contrastive loss term, including: The first complete log-probability of the positive example sequence is constructed using the policy model to be updated and the reference policy model, and the first subsequence log-probability of the negative example sequence is constructed using the self-guided policy model and the reference policy model. A first contrastive loss term is constructed using the difference between the first complete log-log odds and the first subsequence log-log odds.

6. The question-answering method based on self-guided direct preference optimization according to claim 5, characterized in that, Construct the second contrastive loss term, including: The second subsequence log odds of the positive example sequence are constructed using the self-guided strategy model and the reference strategy model, and the second complete log odds of the negative example sequence are constructed using the strategy model to be updated and the reference strategy model. A second contrastive loss term is constructed using the difference between the log-odds of the second subsequence and the log-odds of the second complete sequence.

7. The question-answering method based on self-guided direct preference optimization according to claim 6, characterized in that, The self-guided direct preference optimization loss function is: ； in, The first complete logarithmic probability, The log-probability of the first subsequence. The log-probability of the second subsequence. The second complete logarithmic probability, For the Sigmoid function, For hyperparameters, For expectations; The formulas for obtaining the first complete logarithmic odds, the first subsequence logarithmic odds, the second subsequence logarithmic odds, and the second complete logarithmic odds are as follows: , , and ; in, For the strategy model to be updated, For reference strategy model, This is a self-guided strategy model. This is a positive example sequence. A sequence of positive examples. This is a counterexample sequence. For counterexample sequences, This refers to the cue data used as input to the model in the training data.

8. A question-answering device based on self-guided direct preference optimization, characterized in that, include: The strategy model initialization module is used to determine the initial large language model used to complete the question answering task as the strategy model to be updated, and to initialize the reference strategy model and the self-guided strategy model; wherein, the reference strategy model is used to constrain the log odds of the strategy model to be updated, and the self-guided strategy model is used to regulate the gradient update direction of the strategy model to be updated. The subsequence sampling module is used to obtain positive example sequences of positive example sequences and negative example sequences of negative example sequences from the training data of historical question-answering scenarios. The loss function construction module is used to construct a self-guided direct preference optimization loss function, which includes a first contrastive loss term between the positive example sequence and the negative example sequence and a second contrastive loss term between the positive example sequence and the negative example sequence, using the policy model to be updated, the reference policy model, and the self-guided policy model; wherein the first contrastive loss term and the second contrastive loss term are symmetrical structures. The gradient update module is used to iteratively update the parameters of the policy model to be updated using the self-guided direct preference optimization loss function, so as to determine the target policy model as the target large language model. The question-answering task processing module is used to input the current question-answering task into the target large language model to obtain the task processing result output by the target large language model.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the question-answering method based on self-guided direct preference optimization as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the question-answering method based on self-guided direct preference optimization as described in any one of claims 1 to 7.

Images