Preference optimization DPO sample generation method and device and electronic equipment
By generating sample pairs using different sampling temperatures in the DPO model, determining weights and loss information, and adjusting gradients, the problem of existing models struggling to learn human preference features is solved, resulting in more efficient training and more accurate model output.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-02
- Publication Date
- 2026-04-07
AI Technical Summary
Existing preference optimization DPO methods use the same temperature sampling strategy when generating positive and negative samples, making it difficult for the model to effectively learn human preference features.
The DPO model generates a set of sample pairs. The weights and loss information of the sample pairs are determined using different sampling temperatures. Gradient adjustment is performed to optimize model parameters, including factual consistency and illusion assessment to reduce model error.
It improved the training accuracy and generalization ability of the model, reduced the model illusion rate, and enhanced the model's learning effect.
Smart Images
Figure CN121809568A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of data processing technology, specifically to the fields of artificial intelligence and deep learning, and particularly to a sample generation method, apparatus, and electronic device for preference-optimized DPO. Background Technology
[0002] Direct Preference Optimization (DPO) is a method that directly utilizes preference data to replace the traditional reinforcement learning process. Existing DPO methods use the same temperature sampling strategy when generating positive and negative samples, which makes it difficult for the model to effectively learn human preference features. Summary of the Invention
[0003] This disclosure provides a sample generation method, apparatus, and electronic device for preference-optimized DPO.
[0004] According to one aspect of this disclosure, a sample generation method for preference-optimized DPO is provided, comprising: Confirm task instructions; The DPO model generates a set of sample pairs according to the task instructions. The set of sample pairs includes multiple sample pairs, each sample pair corresponds to two different sampling temperatures, and each sample pair includes a positive sample and a negative sample. Based on the sample pairs and their corresponding two different sampling temperatures, the loss information of the DPO model is determined; Based on the loss information of the DPO model, the model parameters of the DPO model are adjusted by gradient to obtain the target DPO model.
[0005] According to another aspect of this disclosure, a sample generation apparatus for preference-optimized DPO is provided, comprising: The first acquisition module is used to determine the task instructions; The second acquisition module is used to generate a set of sample pairs according to the task instructions through the DPO model. The set of sample pairs includes multiple sample pairs, each sample pair corresponds to two different sampling temperatures, and each sample pair includes a positive sample and a negative sample. The third acquisition module is used to determine the loss information of the DPO model based on the sample pair and its two corresponding different sampling temperatures. The fourth acquisition module is used to perform gradient adjustment on the model parameters of the DPO model based on the loss information of the DPO model, so as to obtain the target DPO model.
[0006] According to a third aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect embodiment.
[0007] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method described in the first aspect embodiment.
[0008] According to a fifth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method described in the first aspect embodiment.
[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description
[0010] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 A schematic diagram illustrating a sample generation method for preference-optimized DPO provided in an embodiment of this disclosure; Figure 2 A schematic diagram illustrating a sample generation method for preference-optimized DPO provided in an embodiment of this disclosure; Figure 3 A schematic diagram illustrating a sample generation method for preference-optimized DPO provided in an embodiment of this disclosure; Figure 4 A structural block diagram of a sample generation device for preference-optimized DPO provided in this application embodiment; Figure 5 A schematic block diagram of an example electronic device used to implement embodiments of the present disclosure is shown. Detailed Implementation
[0011] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.
[0012] Data processing is the collection, storage, retrieval, processing, transformation, and transmission of data. Its basic purpose is to extract and derive valuable and meaningful data from large amounts of potentially messy and difficult-to-understand data.
[0013] Artificial intelligence (AI) is a new technological science that studies and develops theories, methods, technologies, and application systems to simulate, extend, and expand human intelligence. It attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence.
[0014] Deep learning specifically refers to machine learning based on deep neural network models and methods. It has developed based on statistical machine learning, artificial neural network and other algorithmic models, combined with the development of modern big data and high computing power. Its most important technical feature is its ability to automatically extract features. The extracted features are also called deep features or deep feature representations. Compared with manually designed features, deep features have stronger representation capabilities and are more robust.
[0015] Figure 1 This is a schematic diagram illustrating a sample generation method for preference-optimized DPO provided in an embodiment of this disclosure. Figure 1 As shown, the method includes the following steps: S101, confirm task instructions.
[0016] Optionally, task instructions can be input by the user. There can be one or more task instructions, and each task instruction corresponds to a task. For example, the input content is P = {p1, p2, ..., pn}, where P represents the set of input task instructions, and p1, p2, ..., pn are individual task instructions.
[0017] In some embodiments, task instructions may be standardized preprocessed, including but not limited to removing redundant characters and unifying semantic templates, to ensure semantic consistency of tasks.
[0018] S102 generates a set of sample pairs based on task instructions using the DPO model.
[0019] The sample pair set includes multiple sample pairs, each corresponding to two different sampling temperatures, and each sample pair includes a positive sample and a negative sample.
[0020] The core idea of the DPO model is to use human-labeled preference data (such as comparisons of the merits of paired responses) to simplify the original optimization objective into a supervised learning loss function through mathematical derivation, which directly guides the model to generate responses that are more in line with human preferences.
[0021] The DPO model can generate multiple candidate answers of different qualities and initial probabilities based on the task instructions. It can also process the initial probabilities of the candidate answers according to different sampling temperatures to obtain adjusted probabilities. Based on the candidate answers with different adjusted probabilities, it can generate a set of sample pairs and label the positive and negative samples in the sample pairs.
[0022] S103, based on the sample pairs and their corresponding two different sampling temperatures, determine the loss information of the DPO model.
[0023] Optionally, the weights of sample pairs can be determined based on the sample pairs and their corresponding two different sampling temperatures. For example, the weights of sample pairs can be determined based on the difference between the sampling temperatures, thereby dynamically adjusting the contribution of each sample pair, assigning higher weights to high-quality differential samples, and enhancing generalization ability.
[0024] The loss function of the DPO model can be optimized by the probability difference between positive and negative samples. In this embodiment, when determining the loss function corresponding to each sample pair, the loss function can be optimized based on the weight of the sample pair, so that each sample pair in the model is no longer an average value, and more accurate loss information of the DPO model can be obtained.
[0025] S104. Based on the loss information of the DPO model, the model parameters of the DPO model are adjusted by gradient to obtain the target DPO model.
[0026] In this embodiment, the loss information of the DPO model is actually the loss function of the DPO model, which can measure the difference between the predicted value and the true value and quantify the prediction error of the model into a scalar value. The smaller the loss function value, the more accurate the model prediction. The gradient direction of the loss function indicates the optimal direction for parameter adjustment. Based on the gradient direction, the model parameters of the DPO model are adjusted by gradient until the loss function of the DPO model converges, and the final model parameters are obtained, thus obtaining the target DPO model.
[0027] In this embodiment, sample pairs of different qualities are generated according to the task instructions. Based on the sample pairs and the two corresponding different sampling temperatures, the differences between the samples in the sample pairs can be more clearly defined. The weights of the sample pairs are adaptively determined based on the difference between the sampling temperatures of the sample pairs, thereby weighting the loss function corresponding to the sample pairs to obtain more accurate loss information of the DPO model. Based on the more accurate loss information of the DPO model, the model parameters are adjusted by gradient to obtain the target DPO model, thereby improving the training efficiency and training accuracy of the model.
[0028] Figure 2 This is a schematic diagram illustrating a sample generation method for preference-optimized DPO provided in an embodiment of this disclosure. Figure 2As shown, the method includes the following steps: S201, confirm task instructions.
[0029] In this embodiment of the disclosure, the method for implementing step S201 can be implemented in any of the various embodiments of the disclosure, and no limitation is made here, nor will it be described in detail.
[0030] S202 generates a set of sample pairs based on task instructions using the DPO model.
[0031] In this embodiment of the disclosure, the method for implementing step S202 can be implemented in any of the various embodiments of the disclosure, and no limitation is made here, nor will it be described in detail.
[0032] S203 determines the weight of the sample pair based on two different sampling temperatures.
[0033] Optionally, in this embodiment, the two different sampling temperatures are high-temperature sampling and low-temperature sampling, respectively.
[0034] In some embodiments, the temperature difference between two different sampling temperatures can be obtained; based on the temperature difference, the weight of the sample pair is determined, so that the sample with the higher temperature difference has a higher contribution in the optimization process, while the influence of the sample with the lower temperature difference is suppressed; in this embodiment, the weight of the sample pair can be expressed as:
[0035] in, For the first The weights of each sample pair; This represents the temperature difference between the sampled temperatures. , The sampling temperature is the high temperature. This is the low-temperature sampling temperature; Temperature threshold; It is a regulating factor.
[0036] S204. Determine the DPO loss information of the sample pair based on the conditional probability of each sample in the sample pair.
[0037] The conditional probability of each sample is defined as the probability that, given an input x, the model generates the corresponding y, expressed as: ,in, Here are the model parameters, x is the input prompt, and y is the generated response; the conditional probability of each sample represents the probability that the model will generate that response. The DPO adjusts the model by optimizing the ratio of the probabilities of samples in a sample pair to make it generate a response that conforms to preferences.
[0038] Optionally, a temperature adjustment coefficient can be determined based on the temperature difference between two different sampling temperatures. This coefficient controls the impact of the distribution difference between positive and negative samples in a sample pair on the training gradient. In this embodiment, the preference gradient is dynamically adjusted based on the temperature difference between sampling temperatures to improve model training accuracy. The temperature adjustment coefficient is expressed as: ,in The sampling temperature is the high temperature. This is the low-temperature sampling temperature.
[0039] Based on the temperature regulation coefficient and the conditional probability of the samples, the DPO loss information for the sample pairs is determined. In this embodiment, the loss information is the corresponding loss function. The DPO loss information is the loss function of the DPO model, which is expressed as follows:
[0040] in, The loss function for DPO; and These are the conditional probabilities of the sample against the low-temperature sample and the high-temperature sample, respectively. This is the temperature regulation coefficient; The activation function is the Sigmoid function; This represents all sample pairs in the sample pair set. The expected value.
[0041] S205. Determine the loss information of the DPO model based on the weights of the sample pairs and the DPO loss information of the sample pairs.
[0042] Optionally, the DPO loss information corresponding to the current sample pair can be weighted based on the weight of the sample pair to obtain the loss information of the DPO model. When learning the difference in preferences, the gradient strength can be adaptively adjusted to prevent overfitting or unstable convergence.
[0043] In some embodiments, factual consistency and / or illusion assessments can also be performed on sample pairs to obtain factual regularization terms for the sample pairs. The factual regularization terms are used to reflect whether the current sample pair conforms to the facts or whether there is an illusion. For example, the factual consistency and / or illusion assessments of sample pairs can be performed using a pre-trained assessment model. In this embodiment, the value of the factual regularization term is 0 or 1. If the sample pair has an illusion or is inconsistent with the facts, the output of the factual regularization term is 1; otherwise, if the sample pair does not have an illusion or is consistent with the facts, the output of the factual regularization term is 0.
[0044] Furthermore, based on the factual regularization term, the loss information of the DPO model is corrected to obtain the final loss information of the DPO model. In this embodiment, the factual regularization term is used as a penalty term to correct the loss information of the DPO model, reducing the risk of the model generating illusions and propagating false information. The final loss function of the DPO model is expressed as:
[0045] in, This is the final loss function for the DPO model; For the first The weights of each sample pair are used to control the contribution of the sample pair; For the first DPO loss information for each sample pair; It is a factual regular term; This is the regular intensity coefficient.
[0046] S206. Based on the loss information of the DPO model, the model parameters of the DPO model are adjusted by gradient to obtain the target DPO model.
[0047] It is understandable that the purpose of gradient adjustment of the model parameters of the DPO model is to minimize the loss function of the DPO model, thereby obtaining the fitted target DPO model.
[0048] Optionally, this embodiment can use the backpropagation algorithm to update the model parameters. During backpropagation, the gradient contribution of a sample pair is expressed as:
[0049] in, The gradient contribution of the sample pair; For the first The gradient of the DPO function for each sample pair; For the first The weights of each sample pair; The gradient of the factual regularization term; This is the regular intensity coefficient.
[0050] It is understandable that the greater the temperature difference, When the temperature difference approaches 1, the contribution of reliable samples is the greatest; the smaller the temperature difference or the negative value, the greater the contribution. As the value approaches zero, the influence of low-quality samples weakens, and the model automatically focuses on more effective samples with differences in preferences.
[0051] In some embodiments, the process of the optimizer updating model parameters during backpropagation is represented as follows:
[0052] in, These are the model parameters for the next time step; The learning rate; These are the model parameters at the current moment.
[0053] The final parameter set of the target DPO model is the optimal solution obtained by convergence under the combined constraints of weights and factual regularization terms, which is the model parameters corresponding to the minimum loss function. The minimum loss function is expressed as:
[0054] in, The minimum loss function is used, and the corresponding model parameters are the final model parameters. The target DPO model is determined based on the final model parameters.
[0055] In this embodiment, a set of sample pairs is generated by the DPO model according to the task instructions. The weights of the sample pairs are determined based on different sampling temperatures, and the DPO loss information of the sample pairs is determined based on the conditional probability of each sample in the sample pairs. The DPO loss information is weighted according to the weights, and the gradient strength can be adaptively adjusted when learning preference differences, thereby preventing overfitting or convergence instability. At the same time, factual consistency and / or illusion evaluation is performed on the sample pairs to obtain factual regularization terms for the sample pairs. The DPO loss information is corrected based on the factual regularization terms to reduce the risk of the model generating illusions and propagating erroneous information, thus obtaining the final loss information of the DPO model. The model parameters of the DPO model are then adjusted by gradient based on the loss information of the DPO model to obtain the final target DPO model. This improves the training accuracy of the target DPO model and reduces the model illusion rate, resulting in stronger generalization ability and versatility.
[0056] Figure 3 This is a schematic diagram illustrating a sample generation method for preference-optimized DPO provided in an embodiment of this disclosure. Figure 3 As shown, the method includes the following steps: S301, confirm task instructions.
[0057] In this embodiment of the disclosure, the method for implementing step S301 can be implemented in any of the various embodiments of the disclosure, and no limitation is made here, nor will it be described in detail.
[0058] S302 uses the DPO model to generate multiple candidate words and their initial probabilities based on the task instructions.
[0059] Understandably, candidate words are generated based on the task, including words that are highly relevant to the task and words that are not highly relevant to the task. The initial probability of each candidate word is used to represent the probability that the candidate word is relevant to the task.
[0060] For example, assuming the task is "promote a low-sugar yogurt", the generated candidate words may include, but are not limited to, "contains live probiotics", "bread and cereal", "refreshing in summer", and "the weather is nice today". Among them, "contains live probiotics" and "bread and cereal" are more relevant to the task, so their initial probabilities are higher, for example, the initial probability of "contains live probiotics" is 0.2 and the initial probability of "bread and cereal" is 0.15. "refreshing in summer" and "the weather is nice today" are less relevant to the task, so their initial probabilities are lower, for example, the initial probability of "refreshing in summer" is 0.06 and the initial probability of "the weather is nice today" is 0.01.
[0061] S303 determines two different sampling temperatures based on the task.
[0062] Optionally, the task type corresponding to the task can be determined, and the sampling temperature corresponding to the positive sample and the sampling temperature corresponding to the negative sample can be determined according to the task type to generate differentiated sample pairs and improve sample discrimination. Among them, a low sampling temperature can amplify the proportion of high probability words and compress the proportion of low probability words, and high probability words are selected first; a high sampling temperature can compress the proportion of high probability words and amplify the proportion of low probability words, and low probability words are easier to select.
[0063] For example, assuming the task type is a factual question-answering task that requires highly accurate and fact-based answers, then positive samples correspond to low temperature, generating high-confidence, factually strong answers, while negative samples correspond to high temperature, generating answers with natural language but possibly containing minor errors.
[0064] For example, assuming the task type is a creative writing task that requires diverse, novel, and creative questions and answers, then positive samples correspond to high temperature, generating answers that encourage creativity, while negative samples correspond to low temperature, generating overly conservative answers.
[0065] In response to the task type being the first task type, the sampling temperature corresponding to the positive sample is determined in the first temperature range; the sampling temperature corresponding to the negative sample is determined in the second temperature range. The upper limit of the first temperature range is less than the lower limit of the second temperature range, and the sampling temperature corresponding to the positive sample is less than the sampling temperature corresponding to the negative sample. In this embodiment, the first task type can be a task type that requires highly accurate and fact-based answers, that is, the positive sample is at a low temperature and the negative sample is at a high temperature. For example, the first temperature range is 0.2~0.3 and the second temperature range is 0.7~1.0.
[0066] In response to the task type being the second task type, the sampling temperature corresponding to the positive sample is determined in the third temperature range; the sampling temperature corresponding to the negative sample is determined in the fourth temperature range, where the upper limit of the fourth temperature range is less than the lower limit of the third temperature range, and the sampling temperature corresponding to the positive sample is greater than the sampling temperature corresponding to the negative sample. In this embodiment, the second task type can be a question-and-answer session that requires diversity, novelty, and creativity, that is, the positive sample is at a high temperature and the negative sample is at a low temperature. For example, the third temperature range is 0.7~1.0 and the fourth temperature range is 0.2~0.3.
[0067] S304, the initial probability is adjusted based on two different sampling temperatures to obtain the adjusted probability of the candidate word.
[0068] Optionally, the initial probability can be adjusted separately based on two different sampling temperatures. That is, the initial probability can be adjusted by sampling at low temperature and sampling at high temperature. Low temperature sampling and high temperature sampling will "exponentially scale" the initial probability of candidate words to obtain the first adjusted probability and the second adjusted probability of candidate words. Low temperature sampling will increase the initial probability of high probability words and compress the initial probability of low probability words, while high temperature sampling will increase the initial probability of low probability words and compress the initial probability of high probability words.
[0069] S305, Based on the adjusted probabilities of candidate words, generate sample pairs corresponding to the task to obtain a sample pair set.
[0070] Based on the first adjustment probability, a first candidate word is determined from multiple candidate words, and a positive sample is generated based on the first candidate word; based on the second adjustment probability, a second candidate word is determined from multiple candidate words, and a negative sample is generated based on the first candidate word. Each positive sample and negative sample is a sample pair, thus obtaining a set of sample pairs. Combined with the actual task type, corresponding sample pairs are generated, resulting in stronger generalization ability.
[0071] For example, if the task type is the first type of factual question answering, then the positive sample should be the high probability word after low temperature adjustment, and the negative sample should be the low probability word after high temperature adjustment. That is, the first candidate word with the higher first adjustment probability is determined as the positive sample, and the second candidate word with the lower second adjustment probability is determined as the negative sample.
[0072] For example, if the task type is the creative type of the second task, then the positive sample should be the lower probability word after high temperature adjustment, and the negative sample should be the higher probability word after low temperature adjustment. That is, the second candidate word with the lower second adjustment probability is determined as the positive sample, and the first candidate word with the higher first adjustment probability is the negative sample.
[0073] Optionally, the probability of the first adjustment probability and the second adjustment probability among the candidate words can be distinguished based on a probability threshold. For example, words with a probability higher than the threshold are considered high probability, and words with a probability lower than the threshold are considered low probability. No specific limitation is made. Positive and negative samples are determined according to the task type and adjustment probability to obtain sample pairs that better match the preferences, thereby improving the training speed and accuracy of the model.
[0074] S306 determines the weight of a sample pair based on two different sampling temperatures.
[0075] In this embodiment of the disclosure, the method for implementing step S306 can be implemented in any of the various embodiments of the disclosure, and no limitation is made here, nor will it be described in detail.
[0076] S307, determine the DPO loss information of the sample pair based on the conditional probability of each sample in the sample pair.
[0077] In this embodiment of the disclosure, the method for implementing step S307 can be implemented in any of the various embodiments of the disclosure, and no limitation is made here, nor will it be described in detail.
[0078] S308. Determine the loss information of the DPO model based on the weights of the sample pairs and the DPO loss information of the sample pairs.
[0079] In this embodiment of the disclosure, the method for implementing step S308 can be implemented in any of the various embodiments of the disclosure, and no limitation is made here, nor will it be described in detail.
[0080] S309, based on the loss information of the DPO model, performs gradient adjustment on the model parameters of the DPO model to obtain the target DPO model.
[0081] In this embodiment of the disclosure, the method for implementing step S309 can be implemented in any of the various embodiments of the disclosure, and no limitation is made here, nor will it be described in detail.
[0082] In this embodiment, the DPO model generates candidate words and their initial probabilities based on the task instructions. Two different sampling temperatures are determined according to the actual task type, and the initial probabilities are adjusted based on these two temperatures to identify positive and negative samples, thus obtaining sample pairs. Temperature-differentiated sampling improves sample discriminability. Furthermore, the weights of the sample pairs are determined based on the sampling temperatures, and the DPO loss information for each sample pair is determined based on the conditional probability of each sample. The DPO loss information is weighted according to the weights to prevent overfitting or unstable convergence. Factual consistency and / or illusion assessments are performed on the sample pairs to obtain factual regularization terms that correct the DPO loss information, reducing the risk of model illusion and propagation of incorrect information. This yields the final loss information for the DPO model, improving the training accuracy of the target DPO model and reducing the model illusion rate, resulting in stronger generalization ability and versatility.
[0083] Figure 4 This is a structural block diagram of a sample generation device for preference-optimized DPO provided in an embodiment of this application. Figure 4 As shown, the sample generation apparatus 400 for preference optimization DPO includes: The first acquisition module 401 is used to determine the task instruction; The second acquisition module 402 is used to generate a sample pair set according to the task instructions through the DPO model. The sample pair set includes multiple sample pairs, each sample pair corresponds to two different sampling temperatures, and each sample pair includes a positive sample and a negative sample. The third acquisition module 403 is used to determine the loss information of the DPO model based on the sample pair and its two corresponding different sampling temperatures. The fourth acquisition module 404 is used to perform gradient adjustment on the model parameters of the DPO model based on the loss information of the DPO model in order to obtain the target DPO model.
[0084] In some embodiments, the third acquisition module 403 is configured to: The weights of the sample pairs are determined based on two different sampling temperatures. Based on the conditional probability of each sample in the sample pair, determine the DPO loss information of the sample pair; The loss information of the DPO model is determined based on the weights of the sample pairs and the DPO loss information of the sample pairs.
[0085] In some embodiments, the third acquisition module 403 is configured to: Obtain the temperature difference between two different sampling temperatures; The weights of the sample pairs are determined based on the temperature difference.
[0086] In some embodiments, the third acquisition module 403 is configured to: The temperature adjustment coefficient is determined based on the temperature difference between two different sampling temperatures; Based on the temperature regulation coefficient and the conditional probability of the samples, the DPO loss information of the sample pairs is determined.
[0087] In some embodiments, the third acquisition module 403 is further configured to: Perform factual consistency and / or illusion assessments on the sample pairs to obtain factual regularization terms for the sample pairs; The loss information of the DPO model is corrected based on the factual regularization term to obtain the final loss information of the DPO model.
[0088] In some embodiments, the second acquisition module 402 is configured to: The DPO model generates multiple candidate words and their initial probabilities based on the task instructions. Two different sampling temperatures are determined based on the task; The initial probability is adjusted based on two different sampling temperatures to obtain the adjusted probability of the candidate word; Based on the adjusted probabilities of candidate words, sample pairs corresponding to the task are generated to obtain a sample pair set.
[0089] In some embodiments, the second acquisition module 402 is configured to: The initial probabilities are adjusted based on two different sampling temperatures to obtain the first and second adjusted probabilities of the candidate words. Based on the adjusted probabilities of candidate words, sample pairs are generated, including: Based on the first adjustment probability, a first candidate word is determined from multiple candidate words, and a positive sample is generated based on the first candidate word; Based on the second adjustment probability, a second candidate word is determined from multiple candidate words to generate negative samples based on the first candidate word.
[0090] In some embodiments, the second acquisition module 402 is configured to: Determine the task type corresponding to the task; Based on the task type, determine the sampling temperature corresponding to the positive sample and the sampling temperature corresponding to the negative sample.
[0091] In some embodiments, the second acquisition module 402 is configured to: In response to the task type being the first task type, the sampling temperature corresponding to the positive sample is determined in the first temperature range; Determine the sampling temperature corresponding to the negative sample within the second temperature range; Among them, the upper limit of the first temperature range is less than the lower limit of the second temperature range, and the sampling temperature corresponding to the positive sample is less than the sampling temperature corresponding to the negative sample.
[0092] In some embodiments, the second acquisition module is further configured to: In response to the task type being the second task type, the sampling temperature corresponding to the positive sample is determined in the third temperature range; Determine the sampling temperature corresponding to the negative sample within the fourth temperature range; The upper limit of the fourth temperature range is less than the lower limit of the third temperature range, and the sampling temperature corresponding to the positive sample is greater than the sampling temperature corresponding to the negative sample.
[0093] In this embodiment, the DPO model generates candidate words and their initial probabilities based on the task instructions. Two different sampling temperatures are determined according to the actual task type, and the initial probabilities are adjusted based on these two temperatures to identify positive and negative samples, thus obtaining sample pairs. Temperature-differentiated sampling improves sample discriminability. Furthermore, the weights of the sample pairs are determined based on the sampling temperatures, and the DPO loss information for each sample pair is determined based on the conditional probability of each sample. The DPO loss information is weighted according to the weights to prevent overfitting or unstable convergence. Factual consistency and / or illusion assessments are performed on the sample pairs to obtain factual regularization terms that correct the DPO loss information, reducing the risk of model illusion and propagation of incorrect information. This yields the final loss information for the DPO model, improving the training accuracy of the target DPO model and reducing the model illusion rate, resulting in stronger generalization ability and versatility.
[0094] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0095] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.
[0096] Figure 5 A schematic block diagram of an example electronic device for implementing embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.
[0097] like Figure 5As shown, device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 502 or a computer program loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 may also store various programs and data required for the operation of device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.
[0098] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0099] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as the sample generation method for preference-optimized DPO. For example, in some embodiments, the sample generation method for preference-optimized DPO can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the sample generation method for preference-optimized DPO described above can be performed. Alternatively, in other embodiments, computing unit 501 may be configured by any other suitable means (e.g., by means of firmware) to perform a sample generation method for preference-optimized DPO.
[0100] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0101] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0102] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0103] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0104] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.
[0105] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.
[0106] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.
[0107] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.
Claims
1. A sample generation method for preference-optimized DPO, wherein, The method includes: Confirm task instructions; The DPO model generates a set of sample pairs according to the task instructions. The set of sample pairs includes multiple sample pairs, each sample pair corresponds to two different sampling temperatures, and each sample pair includes a positive sample and a negative sample. Based on the sample pairs and their corresponding two different sampling temperatures, the loss information of the DPO model is determined; Based on the loss information of the DPO model, the model parameters of the DPO model are adjusted by gradient to obtain the target DPO model.
2. The method according to claim 1, wherein, The step of determining the loss information of the DPO model based on the sample pair and its two corresponding different sampling temperatures includes: The weights of the sample pairs are determined based on the two different sampling temperatures. Based on the conditional probability of each sample in the sample pair, determine the DPO loss information of the sample pair; The loss information of the DPO model is determined based on the weights of the sample pairs and the DPO loss information of the sample pairs.
3. The method according to claim 2, wherein, Determining the weight of the sample pair based on the two different sampling temperatures includes: Obtain the temperature difference between the two different sampling temperatures; The weights of the sample pairs are determined based on the temperature difference.
4. The method according to claim 2, wherein, The step of determining the DPO loss information of the sample pair based on the conditional probability of each sample in the sample pair includes: The temperature adjustment coefficient is determined based on the temperature difference between the two different sampling temperatures; Based on the temperature regulation coefficient and the conditional probability of the sample, the DPO loss information of the sample pair is determined.
5. The method according to any one of claims 2-4, wherein, The step of determining the loss information of the DPO model based on the weights of the sample pairs and the DPO loss information of the sample pairs further includes: Perform factual consistency and / or illusion assessment on the sample pairs to obtain factual regularization terms for the sample pairs; Based on the factual regularization term, the loss information of the DPO model is corrected to obtain the final loss information of the DPO model.
6. The method according to any one of claims 1-4, wherein, The step of generating a set of sample pairs using the DPO model according to the task instructions includes: The DPO model generates multiple candidate words and their initial probabilities based on the task in the task instruction. The two different sampling temperatures are determined based on the task described; The initial probability is adjusted based on the two different sampling temperatures to obtain the adjusted probability of the candidate word; Based on the adjusted probabilities of the candidate words, sample pairs corresponding to the task are generated to obtain the sample pair set.
7. The method according to claim 6, wherein, The step of adjusting the initial probability according to the two different sampling temperatures to obtain the adjusted probability of the candidate word includes: Based on the two different sampling temperatures, the initial probability is adjusted to obtain the first adjusted probability and the second adjusted probability of the candidate word. The step of generating the sample pairs based on the adjusted probabilities of the candidate words includes: Based on the first adjustment probability, a first candidate word is determined from the plurality of candidate words, and the positive sample is generated based on the first candidate word; Based on the second adjustment probability, a second candidate word is determined from the plurality of candidate words to generate the negative sample based on the first candidate word.
8. The method according to claim 6, wherein, Determining the two different sampling temperatures according to the task includes: Determine the task type corresponding to the task; Based on the task type, determine the sampling temperature corresponding to the positive sample and the sampling temperature corresponding to the negative sample.
9. The method according to claim 8, wherein, The step of determining the sampling temperature corresponding to the positive sample and the sampling temperature corresponding to the negative sample based on the task type includes: In response to the task type being the first task type, the sampling temperature corresponding to the positive sample is determined in the first temperature range; Determine the sampling temperature corresponding to the negative sample within the second temperature range; Wherein, the upper limit of the first temperature range is less than the lower limit of the second temperature range, and the sampling temperature corresponding to the positive sample is less than the sampling temperature corresponding to the negative sample.
10. The method according to claim 8, wherein, The method further includes: In response to the task type being the second task type, the sampling temperature corresponding to the positive sample is determined in the third temperature range; Determine the sampling temperature corresponding to the negative sample within the fourth temperature range; Wherein, the upper limit of the fourth temperature range is less than the lower limit of the third temperature range, and the sampling temperature corresponding to the positive sample is greater than the sampling temperature corresponding to the negative sample.
11. A sample generation apparatus for preference-optimized DPO, comprising: The first acquisition module is used to determine the task instructions; The second acquisition module is used to generate a set of sample pairs according to the task instructions through the DPO model. The set of sample pairs includes multiple sample pairs, each sample pair corresponds to two different sampling temperatures, and each sample pair includes a positive sample and a negative sample. The third acquisition module is used to determine the loss information of the DPO model based on the sample pair and its two corresponding different sampling temperatures. The fourth acquisition module is used to perform gradient adjustment on the model parameters of the DPO model based on the loss information of the DPO model, so as to obtain the target DPO model.
12. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1-10.
13. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-10.
14. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-10.