Question answering method and device based on self-guiding direct preference optimization, equipment and medium

Abstract

The invention discloses a question and answer method and device based on self-guiding direct preference optimization, equipment and a medium, and relates to the technical field of large language models, and the method comprises the steps: taking an initial large language model used for completing a question and answer task as a to-be-updated strategy model, initializing a reference strategy model and a self-guiding strategy model used for regulating and controlling the gradient updating direction of the to-be-updated strategy model; obtaining a subsequence of a positive example sequence and a subsequence of a counter example sequence in the historical question and answer scene; constructing a self-guiding direct preference optimization loss function comprising a first contrast loss item between the positive example sequence and the negative example sequence and a second contrast loss item between the positive example sequence and the negative example sequence by utilizing the model; performing iterative gradient updating on the to-be-updated strategy model by using the function to obtain a target large language model; and inputting the current question and answer task into the target large language model, and obtaining a task processing result output by the target large language model. The human preference response generation capability of the model is improved; and the training stability is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of large language models, and in particular to a question-answering method, apparatus, device, and medium based on self-guided direct preference optimization. Background Art

[0002] Large language models (LLMs) can be used to complete question-answering tasks. For example, if a large model is trained for financial dialogue, it can be deployed in a financial dialogue scenario. The trained model can generate answers that conform to human preferences based on the user's historical investment preferences (conservative / aggressive). If a large model is trained for financial question recommendation, it can be deployed in a financial recommendation application scenario to generate personalized financial product recommendation lists or investment portfolio suggestions based on the user's profile (risk preference, investment period, return target, etc.).

[0003] In large language model training, existing techniques for aligning model output with human preferences primarily rely on reinforcement learning from human feedback (RLHF). This approach comprises three main steps: supervised fine-tuning (SFT), reward learning, and reinforcement learning (RL). However, traditional RLHF suffers from a complex training process due to its heavy reliance on the reward model during the reinforcement learning optimization step. Training a high-quality reward model requires significant manpower and time, and also requires consideration of data quality and diversity. Furthermore, the reward model is susceptible to data bias and noise, leading to unstable performance.

[0004] To simplify the training process, the Direct Preference Optimization (DPO) method was proposed. This method reparameterizes the reward model, mapping the reward function to the optimal policy, eliminating the reward training step and transforming the preference feedback of the online reward model into offline implicit modeling. However, DPO suffers from high computational costs, lengthy generated text, prone to overfitting, limited ability to generate responses that align with human preferences, and unstable effectiveness, making it difficult to fully meet the needs of practical applications.

[0005] In summary, how to improve the model's ability to generate human-preferred responses and enhance training stability is a problem to be solved in this field. Summary of the Invention

[0006] In view of this, the present invention aims to provide a question-answering method, apparatus, device, and medium based on self-guided direct preference optimization to improve the model's ability to generate human-preferred responses and enhance training stability. The specific solution is as follows:

[0007] In a first aspect, the present application discloses a question-answering method based on self-guided direct preference optimization, comprising:

[0008] The initial large language model used to complete the question-answering task is determined as the strategy model to be updated, and the reference strategy model and the self-steering strategy model are initialized; wherein the self-steering strategy model is used to control the gradient update direction of the strategy model to be updated;

[0009] Obtain positive example sequences of positive example sequences and negative example sequences of negative example sequences from training data in historical question-answering scenarios;

[0010] Constructing a self-guided direct preference optimization loss function including a first contrast loss term between the positive example sequence and the negative example sequence and a second contrast loss term between the positive example sequence and the negative example sequence using the to-be-updated policy model, the reference policy model, and the self-guided policy model; wherein the first contrast loss term and the second contrast loss term are symmetrical structures with each other;

[0011] Iteratively gradient updating the parameters of the to-be-updated policy model using the self-guided direct preference optimization loss function to determine the obtained target policy model as a target large language model;

[0012] The current question-answering task is input into the target large language model to obtain the task processing result output by the target large language model.

[0013] Optionally, obtaining a positive example sequence of a positive example sequence and a counterexample sequence of a counterexample sequence from training data in a historical question-answering scenario includes:

[0014] Obtaining training data in a historical question-answering scenario; wherein the training data includes positive and negative example data as model output;

[0015] lemmatize the positive example data and the negative example data respectively to obtain a positive example sequence and a negative example sequence containing each lemma;

[0016] respectively intercepting a positive example sequence and a negative example sequence from the positive example sequence and the negative example sequence;

[0017] Accordingly, the iterative gradient update of the parameters of the to-be-updated policy model using the self-guided direct preference optimization loss function includes:

[0018] The parameters of the strategy model to be updated are gradient updated using the self-guided direct preference optimization loss function to obtain a new strategy model to be updated. If the preset stop iteration condition is not currently met, the process jumps back to the step of intercepting the positive example sequence and the negative example sequence from the positive example sequence and the negative example sequence, respectively.

[0019] Optionally, extracting a positive example sequence and a negative example sequence from the positive example sequence and the negative example sequence respectively includes:

[0020] Determining a first truncation length of the positive example sequence and a second truncation length of the negative example sequence based on the sequence lengths of the positive example sequence and the negative example sequence and a preset subsequence sampling ratio;

[0021] The continuous segments cut out from the positive example sequence based on the first cutout length are determined as the positive example sequence, and the continuous segments cut out from the negative example sequence based on the second cutout length are determined as the negative example sequence.

[0022] Optionally, the step of performing gradient updating on the parameters of the to-be-updated policy model using the self-guided direct preference optimization loss function to obtain a new to-be-updated policy model includes:

[0023] Derivative the self-guided direct preference optimization loss function based on the parameters of the policy model to be updated to obtain a gradient that characterizes the direction and magnitude of the parameter update,

[0024] By back propagation, the parameters of the to-be-updated policy model are updated along the reverse direction of the gradient to minimize the self-guided direct preference optimization loss function, so as to obtain a new to-be-updated policy model.

[0025] Optionally, the reference policy model is used to constrain the logarithmic probability of the policy model to be updated;

[0026] Accordingly, the first contrast loss term is constructed, including:

[0027] Constructing a first complete logarithmic probability of the positive example sequence using the to-be-updated strategy model and the reference strategy model, and constructing a first subsequence logarithmic probability of the counterexample sequence using the bootstrapping strategy model and the reference strategy model;

[0028] A first contrast loss term is constructed using the difference between the first complete log-odds and the first subsequence log-odds.

[0029] Optionally, construct a second contrast loss term, including:

[0030] Constructing a second subsequence logarithmic probability of the positive example sequence using the self-bootstrapping strategy model and the reference strategy model, and constructing a second complete logarithmic probability of the negative example sequence using the strategy model to be updated and the reference strategy model;

[0031] A second contrast loss term is constructed using the difference between the second subsequence log-odds and the second complete log-odds.

[0032] Optionally, the self-guided direct preference optimization loss function is:

[0033] ;

[0034] in, is the first complete log-odds, is the log-odds probability of the first subsequence, is the logarithmic probability of the second subsequence, is the second full log-odds, is the Sigmoid function, is a hyperparameter, For expectations;

[0035] The first complete logarithmic probability, the first subsequence logarithmic probability, the second subsequence logarithmic probability, and the second complete logarithmic probability are obtained by the following formulas:

[0036] 、 、 and ;

[0037] in, is the strategy model to be updated, For the reference strategy model, is the self-bootstrapping strategy model, is a positive sequence, is a positive example sequence, is a counterexample sequence, is a counterexample sequence, Hint data in the training data is used as model input.

[0038] In a second aspect, the present application discloses a question-answering device based on self-guided direct preference optimization, comprising:

[0039] A strategy model initialization module is used to determine the initial large language model used to complete the question-answering task as the strategy model to be updated, and initialize the reference strategy model and the self-steering strategy model; wherein the self-steering strategy model is used to control the gradient update direction of the strategy model to be updated;

[0040] The subsequence sampling module is used to obtain the positive example sequence of the positive example sequence and the counterexample sequence of the counterexample sequence from the training data in the historical question-answering scenario;

[0041] a loss function construction module, configured to construct a self-guided direct preference optimization loss function comprising a first contrast loss term between the positive example sequence and the negative example sequence and a second contrast loss term between the positive example sequence and the negative example sequence using the to-be-updated policy model, the reference policy model, and the self-guided policy model; wherein the first contrast loss term and the second contrast loss term are symmetrical structures with each other;

[0042] A gradient update module, configured to perform iterative gradient updates on the parameters of the to-be-updated policy model using the self-guided direct preference optimization loss function, so as to determine the obtained target policy model as a target large language model;

[0043] The question-answering task processing module is used to input the current question-answering task into the target large language model to obtain the task processing result output by the target large language model.

[0044] In a third aspect, the present application discloses an electronic device, comprising:

[0045] Memory, used to store computer programs;

[0046] A processor is used to execute the computer program to implement the steps of the aforementioned question-answering method based on self-guided direct preference optimization.

[0047] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the steps of the aforementioned question-answering method based on self-guided direct preference optimization are implemented.

[0048] The beneficial effects of the present application are as follows: the present application determines the initial large language model used to complete the question-answering task as the strategy model to be updated, and initializes the reference strategy model and the self-guided strategy model; wherein, the self-guided strategy model is used to regulate the gradient update direction of the strategy model to be updated; obtains the positive example sequence of the positive example sequence and the counter-example sequence of the counter-example sequence from the training data in the historical question-answering scenario; uses the strategy model to be updated, the reference strategy model and the self-guided strategy model to construct a self-guided direct preference optimization loss function including a first contrast loss term between the positive example sequence and the counter-example sequence and a second contrast loss term between the positive example sequence and the counter-example sequence; wherein, the first contrast loss term and the second contrast loss term are symmetrical structures to each other; uses the self-guided direct preference optimization loss function to iteratively gradient update the parameters of the strategy model to be updated to determine the obtained target strategy model as the target large language model; inputs the current question-answering task into the target large language model to obtain the task processing result output by the target large language model. It can be seen that this application achieves precise control of the model training process by taking the initial large language model as the strategy model to be updated, and combining the reference strategy model and the self-guided strategy model. The self-guided strategy model effectively balances the optimization process of the positive example sequence and the negative example sequence by regulating the gradient update direction. By extracting the positive example sequence and the negative example sequence from the historical question and answer data, a symmetrical first contrast loss term and a second contrast loss term are constructed, so that the model can take into account the preference alignment of positive and negative examples at the same time during the training process. This symmetrical loss design avoids the problem of excessive decline in negative example rewards in traditional methods, improves the stability and efficiency of training, and uses self-guided direct preference optimization loss function for iterative gradient update. The final target large language model can handle the current question and answer task more accurately, generate high-quality answers that meet user preferences, simplify the training process, and reduce dependence on the reward model. At the same time, through subsequence sampling and symmetric loss design, the alignment performance and generation effect of the model are significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0050] Figure 1 This is a flow chart of a question-answering method based on self-guided direct preference optimization disclosed in this application;

[0051] Figure 2A gradient flow diagram of a specific DPO for selecting and rejecting rewards disclosed in this application;

[0052] Figure 3 This is a schematic diagram of the structure of a question-answering device based on self-guided direct preference optimization disclosed in this application;

[0053] Figure 4 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0055] Large language models can be used to complete question-answering tasks. For example, if a large model is trained for financial dialogue, the model can be deployed in a financial dialogue scenario. The trained model can generate answers that conform to human preferences based on the user's historical investment preferences (conservative / aggressive). If a large model is used for financial question recommendation, the model can be deployed in a financial recommendation application scenario to generate a personalized list of financial product recommendations or investment portfolio suggestions based on the user's profile (risk preference, investment period, return target, etc.).

[0056] In large language model training, existing techniques for aligning model output with human preferences primarily rely on reinforcement learning based on human feedback. This approach consists of three main steps: supervised fine-tuning, reward learning, and reinforcement learning. However, traditional RLHFs suffer from a complex training process due to their heavy reliance on reward models during the reinforcement learning optimization step. Training high-quality reward models requires significant manpower and time, and also requires consideration of data quality and diversity. Furthermore, reward models are susceptible to data bias and noise, leading to unstable performance.

[0057] To simplify the training process, direct preference optimization (DPO) was proposed. This method reparameterizes the reward model, maps the reward function to the optimal policy, eliminates the reward training step, and transforms the preference feedback of the online reward model into offline implicit modeling. However, DPO suffers from high computational costs, lengthy generated text, prone to overfitting, limited ability to generate responses that align with human preferences, and unstable effectiveness, making it difficult to fully meet the needs of practical applications.

[0058] To this end, this application provides a question-answering solution based on self-guided direct preference optimization to improve the model's ability to generate human preference responses and enhance training stability.

[0059] See also Figure 1As shown, the embodiment of the present application discloses a question-answering method based on self-guided direct preference optimization, comprising:

[0060] Step S11: Determine the initial large language model used to complete the question-answering task as the strategy model to be updated, and initialize the reference strategy model and the self-guided strategy model; wherein the self-guided strategy model is used to regulate the gradient update direction of the strategy model to be updated.

[0061] Build an initial large language model for completing question-and-answer tasks. The initial large language model can specifically be a financial dialogue large model. Then deploy the model in the financial dialogue scenario. The trained model can generate answers that meet human preferences based on the user's historical investment preferences (conservative / aggressive), improve the accuracy and professionalism of customer service responses, reduce the pressure on manual customer service, and ensure that the answers meet financial regulatory requirements through preference alignment optimization. It can also be a financial recommendation large model. Then deploy the model in the financial recommendation application scenario. Based on user portraits (risk preferences, investment period, return targets, etc.), generate personalized financial product recommendation lists or investment portfolio suggestions. Through gradient update optimization, reduce the length deviation of recommendation results, improve information transmission efficiency, improve the conversion rate and user satisfaction of the recommendation system, and reduce the risk of customer churn caused by "wrong recommendations".

[0062] The initial large language model used to complete the question-answering task is determined as the strategy model to be updated , and initialize the reference strategy model and self-bootstrapping strategy models , the self-guided strategy model is used to regulate the gradient update direction of the strategy model to be updated, while the reference strategy model is used to constrain the logarithmic probability of the strategy model to be updated, that is, to constrain the KL divergence.

[0063] Step S12: Obtain a positive example sequence of a positive example sequence and a counterexample sequence of a counterexample sequence from training data in a historical question-answering scenario.

[0064] In this embodiment, obtaining a positive example sequence of a positive example sequence and a counterexample sequence of a counterexample sequence from training data in a historical question-answering scenario includes: obtaining training data in a historical question-answering scenario; wherein the training data includes positive example data and counterexample data as model outputs; tokenizing the positive example data and the counterexample data, respectively, to obtain a positive example sequence and a counterexample sequence containing each word; and intercepting a positive example sequence and a counterexample sequence from the positive example sequence and the counterexample sequence, respectively.

[0065] Obtain training data in the historical Q&A scenario, where the training data specifically includes data as the model input and data as the model output. The data as the model input is the historical Q&A task x (i.e., the input text sequence), such as "User A has some idle funds. How should they be allocated?", "What financial products are suitable for conservative investors?", etc. The data as the model output includes positive example data y that conforms to human preferences w and negative example data y that conforms to human rejection l , that is, the positive example data is the correct sample that conforms to the expectations of the Q&A task, while the negative example data is the opposite. However, the negative example data can help the model avoid wrong directions. For example, for the Q&A task "User A has some idle funds. How should they be allocated?", its positive example data is "①Allocate 30% to low-risk money funds to ensure liquidity; ②Allocate 40% to balanced hybrid funds for long-term appreciation; ③Allocate 30% to high-growth industries, and pay attention to adjusting the proportion according to personal risk tolerance", and its negative example data is "Buy all high-risk and high-appreciation funds, and it will triple next year", etc. So a set of training data is [x, y w , y l . Tokenize the positive example data and negative example data respectively to obtain a positive example sequence and a negative example sequence containing each token. For example, if the positive example data is "Should leave early today", the positive example sequence after tokenization is ["Today", "Should", "Leave", "Early", "Today"]; From the positive example sequence y w , negative example sequence y l respectively intercept the positive example subsequence , negative example subsequence .

[0066] In this embodiment, the step of respectively intercepting the positive example subsequence and the negative example subsequence from the positive example sequence and the negative example sequence includes: determining the first truncation length of the positive example sequence and the second truncation length of the negative example sequence based on the sequence lengths of the positive example sequence and the negative example sequence and a preset subsequence sampling ratio; determining the continuous segment intercepted from the positive example sequence based on the first truncation length as the positive example subsequence, and determining the continuous segment intercepted from the negative example sequence based on the second truncation length as the negative example subsequence

[0067] To simplify the calculation, introduce and . Here x is the input text sequence and y is the complete output text sequence. Let the token sequence length of y be T, y t represents the token at the t-th index, and y<t represents all tokens before the t-th index. Since is a subsequence of y, and considering that can be represented at the token level as , where * belongs to the set , we can express X2=p2Y2 and X1=p1Y1, where p1 and p2 represent the product of the label probability ratios of the remaining labels in sequences X1 and X2 except for the subsequences Y1 and Y2.

[0068] The length of the positive sequence is l1, and the length of the negative sequence is l2. c is the smaller value of l1 and l2. The preset subsequence sampling ratio can be set for the positive sequence and the negative sequence respectively. For example, the preset subsequence sampling ratio of the positive sequence is r1, and the preset subsequence sampling ratio of the negative sequence is r2, then l c ×r1 is the first truncation length of the positive sequence, l c ×r2 is the second truncation length of the negative example sequence, and then the continuous segments intercepted from the positive example sequence based on the first truncation length are determined as the positive example sequence , and determine the continuous segments intercepted from the counterexample sequence based on the second interception length as the counterexample sequence .

[0069] Step S13: Using the to-be-updated strategy model, the reference strategy model, and the self-guided strategy model, a self-guided direct preference optimization loss function is constructed, which includes a first contrast loss term between the positive example sequence and the negative example sequence, and a second contrast loss term between the positive example sequence and the negative example sequence; wherein the first contrast loss term and the second contrast loss term are symmetrical structures to each other.

[0070] Traditional Determined Policy Alignment (DPO) is a widely adopted technique for optimizing the preferences of LLMs. This method stands out because it innovatively utilizes an analytical mapping to transform the reward function into an optimal policy, simplifying the alignment process without directly using the reward model. The cornerstone of DPO lies in its specific transformation, which is mathematically formulated as follows:

[0071] ;Formula (1)

[0072] Among them, Reward function, is the scaling factor, represents the policy inferred from the reward model, Denotes the reference policy. Here, Z(x) is a normalizing constant that ensures that the probabilities are properly scaled. The DPO will prompt instructions for In the case of Than rejection result The probability of being preferred is expressed as follows:

[0073] ;Formula (2)

[0074] Formula (2) quantifies the relative preference between two responses by comparing their relative reward values. In this probabilistic framework, the loss function of DPO (denoted as )as follows:

[0075] ;Formula (3)

[0076] and is defined as follows:

[0077] ;Formula (4)

[0078] in, represents the sigmoid function, is the scaling factor. Equations (3) and (4) thus encapsulate the principle of the Bradley-Terry model, integrating preference data into the learning process. In this way, DPO ensures that the responses of LLMs are consistent with observed human preferences.

[0079] Given selected reward and reject rewards ,about and The partial derivatives of are as follows:

[0080] ;Formula (5)

[0081] ;Formula (6)

[0082] Furthermore, the formula for the ratio of the increase in the probability of a human-preferred response to the decrease in the probability of a human-unpreferred response is:

[0083] ;Formula (7)

[0084] The chosen reward represents an increase in the log probability of the model outputting a human-preferred response. This is a reflection of rewarding preferred responses during training, increasing the probability of the model generating preferred responses. The rejected reward, on the other hand, represents a decrease in the log probability of a human-disliked response. This is a reflection of penalizing unpreferred responses during training, reducing the probability of the model generating unpreferred responses.

[0085] For example Figure 2 The gradient flow diagram of a specific DPO for selected and rejected rewards is shown in the figure. In the traditional DPO, when When the DPO gradient is small, it tends to decrease rapidly. , and for With only minor adjustments, this behavior limits the ability of LLMs to effectively generate highly preferred responses. As DPO optimization proceeds, the selected reward Increase, but refuse reward Reduce, therefore, / <1. According to formula (7), this leads to faster gradient updates for rejection rewards than for selection rewards. The above theoretical framework points out several directions for improving DPO:

[0086] G1: Preventing Reward Denial Rapidly decreases to a very small value, otherwise the selected reward may be stopped A meaningful update.

[0087] G2: Increase the ratio in Equation (7) to allow for more substantial updates to the selected reward, thereby enhancing the ability of LLMs to generate preferred responses.

[0088] These adjustments aim to optimize the optimization process of RLHF and improve the performance of LLM in aligning with human preferences. To achieve the above goals, it is proposed to incorporate an adjusted preference optimization objective, namely the self-guided policy model, into the loss function of the original SGDPO.

[0089] In this embodiment, constructing a first comparative loss term includes: using the to-be-updated strategy model and the reference strategy model to construct a first complete logarithmic probability of the positive example sequence, and using the self-guided strategy model and the reference strategy model to construct a first subsequence logarithmic probability of the counterexample sequence; and constructing a first comparative loss term using the difference between the first complete logarithmic probability and the first subsequence logarithmic probability.

[0090] Using the policy model to be updated , Reference Strategy Model Constructing positive example sequences The first complete log-odds of , and using the bootstrapped strategy model , Reference Strategy Model Constructing a sequence of counterexamples The first contrast loss term is constructed using the difference between the first complete logarithmic probability and the first subsequence logarithmic probability, which encourages the reward of positive examples to be higher than that of subsequence negative examples.

[0091] In this embodiment, constructing a second contrast loss term includes: using the self-bootstrapping strategy model and the reference strategy model to construct a second subsequence logarithmic probability of the positive example sequence, and using the to-be-updated strategy model and the reference strategy model to construct a second complete logarithmic probability of the negative example sequence; and constructing a second contrast loss term using the difference between the second subsequence logarithmic probability and the second complete logarithmic probability.

[0092] Leveraging a self-guided policy model , Reference Strategy Model Constructing positive example sequences The logarithmic probability of the second subsequence of , and using the strategy model to be updated , Reference Strategy Model Constructing a counterexample sequence The second complete logarithmic probability of the second subsequence is constructed by using the difference between the second subsequence logarithmic probability and the second complete logarithmic probability, and the second contrast loss term encourages the subsequence positive example to be higher than the complete negative example.

[0093] In this embodiment, the self-guided direct preference optimization loss function is:

[0094] ;Formula (8)

[0095] in, is the first complete log-odds, is the log-odds probability of the first subsequence, is the logarithmic probability of the second subsequence, is the second full log-odds, is the Sigmoid function, is a hyperparameter, For expectations;

[0096] In this embodiment, the first complete logarithmic probability, the first subsequence logarithmic probability, the second subsequence logarithmic probability, and the second complete logarithmic probability are obtained by the following formulas:

[0097] 、 、 and ;

[0098] in, is the strategy model to be updated, For the reference strategy model, is the self-bootstrapping strategy model, is a positive sequence, is a positive example sequence, is a counterexample sequence, is a counterexample sequence, Hint data in the training data is used as model input.

[0099] First, the initial form of the self-guided direct preference optimization loss function is as follows:

[0100] ;Formula (9)

[0101] in, is defined as follows:

[0102] ;Formula (10)

[0103] In order to simplify the calculation, the logarithmic probability is introduced:

[0104] 、 、 、 .

[0105] Step S14: using the self-guided direct preference optimization loss function to iteratively gradient update the parameters of the strategy model to be updated, so as to determine the obtained target strategy model as the target large language model.

[0106] In this embodiment, the iterative gradient update of the parameters of the strategy model to be updated using the self-guided direct preference optimization loss function includes: gradient updating the parameters of the strategy model to be updated using the self-guided direct preference optimization loss function to obtain a new strategy model to be updated. If the preset stop iteration condition is not currently met, the process jumps back to the step of respectively intercepting the positive example sequence and the negative example sequence from the positive example sequence and the negative example sequence.

[0107] It can be understood that the parameters of the to-be-updated strategy model are gradient updated using the constructed self-guided direct preference optimization loss function to obtain a new to-be-updated strategy model. If the preset stop iteration condition is not currently met, the process jumps again to the step of intercepting the positive example sequence and the negative example sequence from the positive example sequence and the negative example sequence, respectively. The preset stop iteration condition can specifically be that the current number of iterations reaches a preset iteration threshold, or that the convergence of the new to-be-updated strategy model meets the preset convergence condition.

[0108] In this embodiment, the parameters of the strategy model to be updated are gradient updated using the self-guided direct preference optimization loss function to obtain a new strategy model to be updated, including: deriving the self-guided direct preference optimization loss function based on the parameters of the strategy model to be updated to obtain a gradient that characterizes the direction and magnitude of the parameter update, and updating the parameters of the strategy model to be updated along the opposite direction of the gradient through back propagation to minimize the self-guided direct preference optimization loss function to obtain a new strategy model to be updated.

[0109] The self-guided direct preference optimization loss function is differentiated based on the parameters of the policy model to be updated to obtain the gradient that characterizes the direction and magnitude of the parameter update. Through back propagation, the parameters of the policy model to be updated are updated in the opposite direction of the gradient to minimize the self-guided direct preference optimization loss function to obtain a new policy model to be updated. Specifically, the back propagation can be performed by automatic differentiation.

[0110] Step S15: input the current question-answering task into the target large language model to obtain the task processing result output by the target large language model.

[0111] The resulting large language model is the target large language model. A guidance term is introduced into the traditional loss function to guide the training gradient flow during the optimization process and control the update of the selected reward and the rejected reward. A loss function (i.e., a self-guided direct preference optimization loss function) is constructed based on the objective function. The loss function includes calculation terms related to the guidance strategy model and is used to adjust the parameter updates of the model. Subsequences are constructed from the original sequence, and by adjusting the length-related hyperparameters of the subsequences, the gradient changes of the selected reward and the rejected reward are affected to optimize model training.

[0112] Next, the target large language model can be used to process the current question-answering task, and then the task processing result output by the large model can be obtained, and the task processing result is in line with human preferences.

[0113] The beneficial effects of the present application are as follows: the present application determines the initial large language model used to complete the question-answering task as the strategy model to be updated, and initializes the reference strategy model and the self-guided strategy model; wherein, the self-guided strategy model is used to regulate the gradient update direction of the strategy model to be updated; obtains the positive example sequence of the positive example sequence and the counter-example sequence of the counter-example sequence from the training data in the historical question-answering scenario; uses the strategy model to be updated, the reference strategy model and the self-guided strategy model to construct a self-guided direct preference optimization loss function including a first contrast loss term between the positive example sequence and the counter-example sequence and a second contrast loss term between the positive example sequence and the counter-example sequence; wherein, the first contrast loss term and the second contrast loss term are symmetrical structures to each other; uses the self-guided direct preference optimization loss function to iteratively gradient update the parameters of the strategy model to be updated to determine the obtained target strategy model as the target large language model; inputs the current question-answering task into the target large language model to obtain the task processing result output by the target large language model. It can be seen that this application achieves precise control of the model training process by taking the initial large language model as the strategy model to be updated, and combining the reference strategy model and the self-guided strategy model. The self-guided strategy model effectively balances the optimization process of the positive example sequence and the negative example sequence by regulating the gradient update direction. By extracting the positive example sequence and the negative example sequence from the historical question and answer data, a symmetrical first contrast loss term and a second contrast loss term are constructed, so that the model can take into account the preference alignment of positive and negative examples at the same time during the training process. This symmetrical loss design avoids the problem of excessive decline in negative example rewards in traditional methods, improves the stability and efficiency of training, and uses self-guided direct preference optimization loss function for iterative gradient update. The final target large language model can handle the current question and answer task more accurately, generate high-quality answers that meet user preferences, simplify the training process, and reduce dependence on the reward model. At the same time, through subsequence sampling and symmetric loss design, the alignment performance and generation effect of the model are significantly improved.

[0114] The self-guided direct preference optimization loss function in this embodiment has great advantages in enhancing preference optimization. The following is the content proving the advantages of the self-guided direct preference optimization loss function:

[0115] first, about and The partial derivatives of are as follows:

[0116] ;Formula (11)

[0117] From formula (11), we can see that and The gradients of depend on Y1 and Y2 respectively. Therefore, by manipulating Y1 and Y2, we can control the gradient flow in the alignment method, thereby affecting the updates of the selected and rejected rewards.

[0118] In addition, for The formula is as follows:

[0119] ;Formula (12)

[0120] In addition, for The formula is as follows:

[0121] ;Formula (13)

[0122] Secondly, the partial derivative As Y2 increases, the partial derivative increases. As Y1 increases, it decreases. During fine-tuning, the preference alignment algorithm increases the probability of generating preferred text while decreasing the probability of generating non-preferred text, so p2<1. Therefore, Y2>X2. Comparing Formula 11 with Formula 5 and Formula 6, we can see that the only difference lies in one variable. For example, X2 in Formula 5 is replaced by Y2 in Formula 11. This results in the following formula:

[0123] ;Formula (14)

[0124] Formula (14) shows that SGDPO can alleviate the rapid decline of rejection rewards. Obviously, as the model converges, both the selected rewards and the rejection rewards maintain higher values ​​compared to the DPO baseline. Therefore, SGDPO enhances the update of the selected rewards, thereby promoting the generation of preferred responses.

[0125] Among them, for The formula is as follows:

[0126] ;Formula (15)

[0127] In addition, for The formula is as follows:

[0128] ;Formula (16)

[0129] Next, set = , z=Y1 / Y2, for each pairwise preference instance, the ratio of the increase in the probability of a human preference response to the decrease in the probability of a human non-preference response is:

[0130] ;Formula (17)

[0131] Among them, The following definitions are given:

[0132] ;Formula (18)

[0133] f(z) is a monotonically increasing function with respect to z. When p1p2<1, f(z) increases as z increases; conversely, when p1p2>1, f(z) decreases as z increases. Furthermore, when p1p2<1, f(z)>1. If the rejection reward decreases rapidly, it will cause p1p2<1. This will make f(z)>1, thereby improving the ratio given by Formula (17). In addition, during training, as z=y1 / y2 increases, f(z) also increases. This behavior is consistent with the goal G2, thereby enhancing the ability of LLM to generate preferred text.

[0134] Among them, for the above content, there are the following formulas:

[0135] ;Formula (19)

[0136] Setting X2=p2Y2 and X1=p1Y1, z=y1 / y2, we have the following formula:

[0137] ;Formula (20)

[0138] In addition: The derivative of f(z) with respect to z is:

[0139] ;Formula (21)

[0140] Since z=y1 / y2, does 、 It depends on the values ​​of p1 and p2. Therefore, when p1p2<1, the function f(z) is monotonically increasing. Conversely, when p1p2>1, the function f(z) is monotonically decreasing.

[0141] In summary, from the sequence y w and y l Derived subsequence and . and As an indicator, it is used to guide the update refinement of the selected reward and rejected reward adjustment. Let l1 and l2 be the sequence y w and y l The length of l c Defined as the smaller value of l1 and l2. w ,y l ) in the pair, randomly select a length (l c ×r1,l c ×r2) preference data pairs ( , ̂), where r1 and r2 are hyperparameters. Adjusting r1 and r2 can affect the gradient changes associated with the selected and rejected rewards during the preference optimization process. For example, decreasing r2 will cause r1 to increase because it is derived from samples with rejected rewards, which will further enhance the update amplitude of the selected reward.

[0142] See also Figure 3 As shown, the embodiment of the present application discloses a question-answering device based on self-guided direct preference optimization, comprising:

[0143] A strategy model initialization module 11 is used to determine the initial large language model used to complete the question-answering task as the strategy model to be updated, and initialize the reference strategy model and the self-guided strategy model; wherein the self-guided strategy model is used to control the gradient update direction of the strategy model to be updated;

[0144] A subsequence sampling module 12 is used to obtain a positive example sequence of a positive example sequence and a negative example sequence of a negative example sequence from training data in a historical question-answering scenario;

[0145] A loss function construction module 13 is configured to construct a self-guided direct preference optimization loss function comprising a first contrast loss term between the positive example sequence and the negative example sequence and a second contrast loss term between the positive example sequence and the negative example sequence using the to-be-updated policy model, the reference policy model, and the self-guided policy model; wherein the first contrast loss term and the second contrast loss term are symmetrical structures with each other;

[0146] A gradient updating module 14 is configured to perform iterative gradient updates on the parameters of the to-be-updated policy model using the self-guided direct preference optimization loss function, so as to determine the obtained target policy model as a target large language model;

[0147] The question-answering task processing module 15 is used to input the current question-answering task into the target large language model to obtain the task processing result output by the target large language model.

[0148] The beneficial effects of the present application are as follows: the present application determines the initial large language model used to complete the question-answering task as the strategy model to be updated, and initializes the reference strategy model and the self-guided strategy model; wherein, the self-guided strategy model is used to regulate the gradient update direction of the strategy model to be updated; obtains the positive example sequence of the positive example sequence and the counter-example sequence of the counter-example sequence from the training data in the historical question-answering scenario; uses the strategy model to be updated, the reference strategy model and the self-guided strategy model to construct a self-guided direct preference optimization loss function including a first contrast loss term between the positive example sequence and the counter-example sequence and a second contrast loss term between the positive example sequence and the counter-example sequence; wherein, the first contrast loss term and the second contrast loss term are symmetrical structures to each other; uses the self-guided direct preference optimization loss function to iteratively gradient update the parameters of the strategy model to be updated to determine the obtained target strategy model as the target large language model; inputs the current question-answering task into the target large language model to obtain the task processing result output by the target large language model. It can be seen that this application achieves precise control of the model training process by taking the initial large language model as the strategy model to be updated, and combining the reference strategy model and the self-guided strategy model. The self-guided strategy model effectively balances the optimization process of the positive example sequence and the negative example sequence by regulating the gradient update direction. By extracting the positive example sequence and the negative example sequence from the historical question and answer data, a symmetrical first contrast loss term and a second contrast loss term are constructed, so that the model can take into account the preference alignment of positive and negative examples at the same time during the training process. This symmetrical loss design avoids the problem of excessive decline in negative example rewards in traditional methods, improves the stability and efficiency of training, and uses self-guided direct preference optimization loss function for iterative gradient update. The final target large language model can handle the current question and answer task more accurately, generate high-quality answers that meet user preferences, simplify the training process, and reduce dependence on the reward model. At the same time, through subsequence sampling and symmetric loss design, the alignment performance and generation effect of the model are significantly improved.

[0149] Furthermore, an embodiment of the present application also provides an electronic device. Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram should not be considered as any limitation to the scope of application of the present application.

[0150] Figure 4This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Specifically, the device may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps of the question-answering method based on self-guided direct preference optimization performed by the electronic device as disclosed in any of the aforementioned embodiments.

[0151] In this embodiment, the power supply 23 is used to provide operating voltage for various hardware devices on the electronic device; the communication interface 24 can create a data transmission channel between the electronic device and external devices. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0152] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0153] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon include an operating system 221, a computer program 222 and data 223, etc. The storage method can be temporary storage or permanent storage.

[0154] The operating system 221 is used to manage and control the hardware devices and computer program 222 on the electronic device, so as to enable the processor 21 to calculate and process the massive amount of data 223 in the memory 22. It can be Windows, Unix, Linux, etc. In addition to including computer programs capable of implementing the question-answering method based on self-guided direct preference optimization and executed by the electronic device as disclosed in any of the aforementioned embodiments, the computer program 222 can further include computer programs capable of performing other specific tasks. In addition to data received by the electronic device and transmitted from external devices, the data 223 can also include data collected by its own input and output interface 25.

[0155] Furthermore, this application discloses a computer-readable storage medium for storing a computer program; wherein, when executed by a processor, the computer program implements the aforementioned question-answering method based on self-guided direct preference optimization. The specific steps of this method can be found in the corresponding content disclosed in the aforementioned embodiments and will not be repeated here.

[0156] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0157] Professionals may further appreciate that the units and algorithmic steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. The steps of the method or algorithm described in conjunction with the embodiments disclosed herein can be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in random access memory (RAM), memory, read-only memory (ROM), electrically programmable EPROM (Erasable Programmable Read Only Memory), electrically erasable programmable EEPROM (Electrically Erasable Programmable read only memory), registers, hard disk, removable disk, CD-ROM (Compact Disc Read-Only Memory), or any other form of storage medium known in the technical field.

[0158] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0159] The above is a detailed introduction to the question-answering method, device, equipment and medium based on self-guided direct preference optimization provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A question-answering method based on self-guided direct preference optimization, characterized in that: include: The initial large language model used to complete the question-answering task is determined as the strategy model to be updated, and the reference strategy model and the self-steering strategy model are initialized; wherein the self-steering strategy model is used to control the gradient update direction of the strategy model to be updated; Obtain positive example sequences of positive example sequences and negative example sequences of negative example sequences from training data in historical question-answering scenarios; Constructing a self-guided direct preference optimization loss function including a first contrast loss term between the positive example sequence and the negative example sequence and a second contrast loss term between the positive example sequence and the negative example sequence using the to-be-updated policy model, the reference policy model, and the self-guided policy model; wherein the first contrast loss term and the second contrast loss term are symmetrical structures with each other; Iteratively gradient updating the parameters of the to-be-updated policy model using the self-guided direct preference optimization loss function to determine the obtained target policy model as a target large language model; The current question-answering task is input into the target large language model to obtain the task processing result output by the target large language model.

2. The question-answering method based on self-guided direct preference optimization according to claim 1, characterized in that: The step of obtaining a positive example sequence of a positive example sequence and a counterexample sequence of a counterexample sequence from training data in a historical question-answering scenario includes: Obtaining training data in a historical question-answering scenario; wherein the training data includes positive and negative example data as model output; lemmatize the positive example data and the negative example data respectively to obtain a positive example sequence and a negative example sequence containing each lemma; respectively intercepting a positive example sequence and a negative example sequence from the positive example sequence and the negative example sequence; Accordingly, the iterative gradient update of the parameters of the to-be-updated policy model using the self-guided direct preference optimization loss function includes: The parameters of the strategy model to be updated are gradient updated using the self-guided direct preference optimization loss function to obtain a new strategy model to be updated. If the preset stop iteration condition is not currently met, the process jumps back to the step of intercepting the positive example sequence and the negative example sequence from the positive example sequence and the negative example sequence, respectively.

3. The question-answering method based on self-guided direct preference optimization according to claim 2, characterized in that: The extracting the positive example sequence and the negative example sequence from the positive example sequence and the negative example sequence respectively includes: Determining a first truncation length of the positive example sequence and a second truncation length of the negative example sequence based on the sequence lengths of the positive example sequence and the negative example sequence and a preset subsequence sampling ratio; The continuous segments cut out from the positive example sequence based on the first cutout length are determined as the positive example sequence, and the continuous segments cut out from the negative example sequence based on the second cutout length are determined as the negative example sequence.

4. The question-answering method based on self-guided direct preference optimization according to claim 2, characterized in that: The step of using the self-guided direct preference optimization loss function to perform gradient updating on the parameters of the to-be-updated policy model to obtain a new to-be-updated policy model includes: Derivative the self-guided direct preference optimization loss function based on the parameters of the policy model to be updated to obtain a gradient representing the direction and magnitude of the parameter update, By back propagation, the parameters of the to-be-updated policy model are updated along the reverse direction of the gradient to minimize the self-guided direct preference optimization loss function, so as to obtain a new to-be-updated policy model.

5. The question-answering method based on self-guided direct preference optimization according to any one of claims 1 to 4, characterized in that: The reference strategy model is used to constrain the logarithmic probability of the strategy model to be updated; Accordingly, the first contrast loss term is constructed, including: Constructing a first complete logarithmic probability of the positive example sequence using the to-be-updated strategy model and the reference strategy model, and constructing a first subsequence logarithmic probability of the counterexample sequence using the bootstrapping strategy model and the reference strategy model; A first contrast loss term is constructed using the difference between the first complete log-odds and the first subsequence log-odds.

6. The question-answering method based on self-guided direct preference optimization according to claim 5, characterized in that: Construct the second contrast loss term, including: Constructing a second subsequence logarithmic probability of the positive example sequence using the self-bootstrapping strategy model and the reference strategy model, and constructing a second complete logarithmic probability of the negative example sequence using the strategy model to be updated and the reference strategy model; A second contrast loss term is constructed using the difference between the second subsequence log-odds and the second complete log-odds.

7. The question-answering method based on self-guided direct preference optimization according to claim 6, characterized in that: The self-guided direct preference optimization loss function is: ； in, is the first complete log-odds, is the log-odds probability of the first subsequence, is the logarithmic probability of the second subsequence, is the second full log-odds, is the Sigmoid function, is a hyperparameter, For expectations; The first complete logarithmic probability, the first subsequence logarithmic probability, the second subsequence logarithmic probability, and the second complete logarithmic probability are obtained by the following formulas: 、 、 and ; in, is the strategy model to be updated, For the reference strategy model, is the self-bootstrapping strategy model, is a positive sequence, is a positive example sequence, is a counterexample sequence, is a counterexample sequence, Hint data in the training data is used as model input.

8. A question-answering device based on self-guided direct preference optimization, characterized in that: include: A strategy model initialization module is used to determine the initial large language model used to complete the question-answering task as the strategy model to be updated, and initialize the reference strategy model and the self-steering strategy model; wherein the self-steering strategy model is used to control the gradient update direction of the strategy model to be updated; The subsequence sampling module is used to obtain the positive example sequence of the positive example sequence and the counterexample sequence of the counterexample sequence from the training data in the historical question-answering scenario; a loss function construction module, configured to construct a self-guided direct preference optimization loss function comprising a first contrast loss term between the positive example sequence and the negative example sequence and a second contrast loss term between the positive example sequence and the negative example sequence using the to-be-updated policy model, the reference policy model, and the self-guided policy model; wherein the first contrast loss term and the second contrast loss term are symmetrical structures with each other; A gradient update module, configured to perform iterative gradient updates on the parameters of the to-be-updated policy model using the self-guided direct preference optimization loss function, so as to determine the obtained target policy model as a target large language model; The question-answering task processing module is used to input the current question-answering task into the target large language model to obtain the task processing result output by the target large language model.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the question-answering method based on self-guided direct preference optimization as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Used to store a computer program; wherein, when the computer program is executed by a processor, the steps of the question-answering method based on self-guided direct preference optimization are implemented as described in any one of claims 1 to 7.

Images