Data processing methods, apparatus, computer equipment, and storage media
By adjusting the learning method of a domain-specific large model during training using the difference scores and reward values of training samples, the problem of poor model inference performance is solved, and response text generation that is more in line with human preferences is achieved.
Patent Information
- Application Number
- CN202611467666.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-20
- Publication Date
- 2026-11-17
AI Technical Summary
Existing domain-specific large models learn the knowledge contained in multiple training samples equally during the reinforcement learning alignment process, resulting in poor inference performance of the trained models.
By acquiring the difference scores and reward values of training samples, the learning method of the domain-wide model during training can be adjusted to learn knowledge from different training samples in a differentiated manner, thereby optimizing the training process to improve the model's response text to better align with human preferences.
It improves the reasoning performance of large domain models, making the generated response text more in line with human preferences.
Smart Images

Figure CN123115352A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer technology, and in particular to a data processing method, apparatus, computer equipment, and storage medium. Background Technology
[0002] Domain-wide models can output response text to input prompt text. As domain-wide models evolve, after supervised fine-tuning, reinforcement learning alignment is typically performed on the fine-tuned models to ensure that the response text output by the domain-wide models better aligns with user preferences.
[0003] Currently, reinforcement learning alignment methods include: acquiring multiple training samples, each training sample including a prompt text and two response texts for that prompt text, with users having different preferences for the two response texts; and training a supervised fine-tuned domain model using Direct Preference Optimization (DPO) based on multiple training samples.
[0004] During DPO training, the domain-wide model learns the knowledge contained in multiple training samples equally. However, since the multiple training samples are different and contain different knowledge, the domain-wide model's inference performance will be poor due to the fact that it learns the knowledge contained in multiple training samples equally. Summary of the Invention
[0005] This disclosure provides a data processing method, apparatus, computer device, and storage medium to at least address the problem of poor inference performance in large domain-specific models trained in related technologies. The technical solution of this disclosure is as follows: According to one aspect of the embodiments of this disclosure, a data processing method is provided, including: Obtain training samples and the corresponding difference scores for the training samples. The training samples include a first response text and a second response text for the same prompt text. The difference scores represent the user's preference difference for the first response text and the second response text. Based on the training samples and the domain big model, the reward value corresponding to the training samples is obtained, and the domain big model is used to output the response text for the input prompt text; Based on the difference scores and reward values corresponding to the training samples, the domain-wide model is trained using direct preference optimization.
[0006] Optionally, the step of performing direct preference optimization training on the domain-wide model based on the difference scores and reward values corresponding to the training samples includes: Based on the difference score and reward value corresponding to the training sample, determine the loss value corresponding to the training sample; Based on the loss values corresponding to the training samples, the domain-wide model is trained using direct preference optimization.
[0007] Optionally, determining the loss value corresponding to the training sample based on the difference score and reward value corresponding to the training sample includes: Based on the target coefficient value of the smoothing coefficient and the difference score, the weights corresponding to the training samples are determined. The weights indicate the reliability of the training samples, and the smoothing coefficient is used to control the smoothness of the weights as the difference score changes. Based on the weights and reward values corresponding to the training samples, the loss value corresponding to the training samples is determined.
[0008] Optionally, determining the weights corresponding to the training samples based on the target coefficient value of the smoothing coefficient and the difference score includes: Based on the target coefficient value, the difference score is scaled to obtain the scaled difference score; The weights are obtained by non-linearly mapping the scaled difference scores using an activation function.
[0009] Optionally, the target coefficient value varies with the difference score.
[0010] Optionally, the method further includes: Based on the difference score and reward value corresponding to the training sample, the target coefficient value is obtained from multiple candidate coefficient values of the smoothing coefficient. The target coefficient value is the candidate coefficient value that minimizes the loss value.
[0011] Optionally, there are multiple training samples, and each training sample corresponds to a loss value and a weight; The step of performing direct preference optimization training on the domain-wide model based on the loss values corresponding to the training samples includes: The total loss value is obtained based on the loss values and weights corresponding to multiple training samples; Based on the total loss value, the model parameters of the large domain model are updated.
[0012] Optionally, obtaining the total loss value based on the loss values and weights corresponding to multiple training samples includes: Based on the weights corresponding to the multiple training samples, the loss values corresponding to the multiple training samples are weighted and averaged to obtain the total loss value.
[0013] Optionally, the process of obtaining the difference score includes: Based on the preference score corresponding to the first response text and the preference score corresponding to the second response text, the difference score is obtained, and the preference score represents the degree of preference of the user for the corresponding response text.
[0014] According to another aspect of the embodiments of this disclosure, a data processing apparatus is provided, comprising: The first acquisition unit is configured to acquire training samples and the difference scores corresponding to the training samples. The training samples include a first response text and a second response text for the same prompt text. The difference scores represent the user's preference difference for the first response text and the second response text. The second acquisition unit is configured to perform an operation based on the training samples and the domain big model to acquire the reward value corresponding to the training samples, wherein the domain big model is used to output response text for the input prompt text. The training unit is configured to perform direct preference optimization training on the domain large model based on the difference scores and reward values corresponding to the training samples.
[0015] Optionally, the training unit includes: The subunit is configured to perform the determination of the loss value corresponding to the training sample based on the difference score and reward value corresponding to the training sample. The training subunit is configured to perform direct preference optimization training on the domain-wide model based on the loss values corresponding to the training samples.
[0016] Optionally, the determining subunit includes: The weight determination subunit is configured to perform a target coefficient value based on a smoothing coefficient and the difference score to determine the weights corresponding to the training samples, wherein the weights indicate the reliability of the training samples, and the smoothing coefficient is used to control the smoothness of the weights as the difference score changes. The loss determination subunit is configured to determine the loss value corresponding to the training sample based on the weight and reward value corresponding to the training sample.
[0017] Optionally, the weight determination subunit is configured to perform: Based on the target coefficient value, the difference score is scaled to obtain the scaled difference score; The weights are obtained by non-linearly mapping the scaled difference scores using an activation function.
[0018] Optionally, the target coefficient value varies with the difference score.
[0019] Optionally, the data processing apparatus further includes: The third acquisition unit is configured to perform an operation based on the difference score and reward value corresponding to the training sample to acquire the target coefficient value from a plurality of candidate coefficient values of the smoothing coefficient, wherein the target coefficient value is a candidate coefficient value that minimizes the loss value.
[0020] Optionally, there are multiple training samples, and each training sample corresponds to a loss value and a weight; The training subunit includes: The sub-unit is configured to perform a total loss value based on the loss values and weights corresponding to multiple training samples. The update subunit is configured to update the model parameters of the domain large model based on the total loss value.
[0021] Optionally, the sub-unit is configured to perform a weighted average of the loss values corresponding to the plurality of training samples based on the weights corresponding to the plurality of training samples to obtain the total loss value.
[0022] Optionally, the first acquisition unit is configured to acquire the difference score based on the preference score corresponding to the first response text and the preference score corresponding to the second response text, wherein the preference score represents the degree of preference of the user for the corresponding response text.
[0023] According to another aspect of the embodiments of this disclosure, a computer device is provided, comprising: One or more processors; One or more memories for storing the one or more processor-executable instructions; The one or more processors are configured to perform a data processing method in any of the possible implementations of the above-described aspects.
[0024] According to another aspect of the present disclosure, a computer-readable storage medium is provided such that, when at least one instruction in the computer-readable storage medium is executed by one or more processors of a computer device, the computer device is enabled to perform a data processing method in any possible implementation of the above aspect.
[0025] According to another aspect of the present disclosure, a computer program product is provided, including one or more instructions that can be executed by one or more processors of a computer device, enabling the computer device to perform a data processing method in any possible implementation of the above aspect.
[0026] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects: By using the difference scores corresponding to the training samples to represent the user's preference differences between different response texts for the same prompt text, the reward value and difference score corresponding to the training samples are comprehensively considered during the direct preference optimization training of the domain big model. This enables the domain big model to learn the knowledge contained in different training samples in a differentiated manner based on the difference scores, so that the response text generated by the domain big model is more in line with human preferences, thereby improving the reasoning performance of the domain big model.
[0027] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description
[0028] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.
[0029] Figure 1 This is a schematic diagram illustrating the implementation environment of a data processing method according to an exemplary embodiment; Figure 2 This is a flowchart illustrating a data processing method according to an exemplary embodiment; Figure 3 This is a flowchart illustrating another data processing method according to an exemplary embodiment; Figure 4 This is a flowchart illustrating an improved DPO training process according to an exemplary embodiment; Figure 5 This is a comparison chart of loss changes during a training process, according to an exemplary embodiment. Figure 6 This is a comparison chart of accuracy changes in a domain-wide model according to an exemplary embodiment; Figure 7 This is an evaluation comparison chart in a VL multimodal scenario, illustrated according to an exemplary embodiment. Figure 8 This is a comparison chart of evaluations in an LLM scenario, illustrated according to an exemplary embodiment. Figure 9 This is a logical structure block diagram of a data processing apparatus according to an exemplary embodiment; Figure 10 This is a schematic diagram illustrating the structure of a computer device according to an exemplary embodiment. Detailed Implementation
[0030] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.
[0031] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.
[0032] The information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by the parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.
[0033] In some embodiments, the meaning of A and / or B includes three cases: A and B, and A and B.
[0034] Figure 1 This is a schematic diagram illustrating the implementation environment of a data processing method according to an exemplary embodiment. See also: Figure 1 In this implementation environment, a terminal 101 and a server 102 may be included. The terminal 101 can be connected to the server 102 via a wireless network or a wired network.
[0035] Terminal 101 can be at least one of a smartphone, wearable device, desktop computer, tablet computer, laptop computer, etc. Terminal 101 can refer to one of multiple terminals; this embodiment uses terminal 101 as an example. Those skilled in the art will understand that the number of terminals can be more or less. For example, there can be several terminals, or dozens or hundreds of terminals, or even more. This disclosure does not limit the number or type of terminals.
[0036] Server 102 can be at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Optionally, the number of servers can be more or fewer, and this disclosure does not limit this. Of course, server 102 may also include other functional servers to provide more comprehensive and diversified services.
[0037] In some embodiments, an application may be installed and run on terminal 101, which can perform model training for large-scale domain models. In this deployment mode, the application can be associated with server 102, which provides background services to terminal 101. For example, during the running of the application, terminal 101 performs model training based on the computation results pushed by the application's background server 102.
[0038] Domain-specific large models refer to large models obtained by pre-training a basic general-purpose large model using business data from a specific domain. These models absorb the professional terminology, business rules, and domain knowledge of their respective domains, optimizing their capabilities for tasks within that domain. Compared to general-purpose large models, they exhibit higher professionalism and accuracy in tasks such as question answering, reasoning, and content generation within a limited domain. They are primarily designed for use in specific industries or niche business scenarios and do not aim for universal capabilities across various scenarios. The input to a domain-specific large model is a prompt text, which guides the model to generate a response text. The response text is a reply or response to the prompt text; that is, the domain-specific large model can output a response text to the input prompt text, which is used to answer the prompt text.
[0039] The domain-specific large model involved in this disclosure can be a domain vertical large model, or it can be a domain-specific large model of other types. Here, the specific classification of the domain-specific large model is not limited in the embodiments of this disclosure. The domain-specific large model can be any of the following: Language Model (LM), Large Language Model (LLM), or Multimodal Language Model. Here, the model type of the domain-specific large model is not limited in the embodiments of this disclosure.
[0040] In some embodiments, an application can be installed and run on server 102, which can perform model training. In this deployment mode, the application can be associated with terminal 101. For example, terminal 101 can serve as a user interaction interface to receive user operation commands, upload training samples, and display training results.
[0041] In some embodiments, server 102 undertakes the primary computing task, and terminal 101 undertakes the secondary computing task; or, server 102 undertakes the secondary computing task, and terminal 101 undertakes the primary computing task; or, server 102 and terminal 101 collaborate on computing using a distributed computing architecture. This disclosure will not elaborate further on these aspects.
[0042] It should be noted that the above implementation environment is only an example, and the method provided in this embodiment can also be executed in other implementation environments. This embodiment does not limit this.
[0043] This disclosure provides a novel DPO training mechanism for training large domain models after supervised fine-tuning (SFT) using DPO, thereby achieving reinforcement learning alignment. The following is a combination of... Figure 2 and Figure 3 This section will introduce the mechanism.
[0044] Figure 2 This is a flowchart illustrating a data processing method according to an exemplary embodiment, see [link to flowchart]. Figure 2 This data processing method is applied to a computer device, which can be either the terminal 101 mentioned above or the server 102 mentioned above. For example... Figure 2 As shown, the data processing method includes steps 201 to 203.
[0045] In step 201, the computer device acquires training samples and the corresponding difference scores for the training samples. The training samples include a first response text and a second response text for the same prompt text. The difference scores represent the user's preference difference for the first response text and the second response text.
[0046] There is at least one training sample. The prompt text can serve as input to the domain-wide model, guiding it to generate response text. The response text is a reply or response to the prompt text.
[0047] The computer device acquires a training set, which includes at least one training sample and a difference score for each training sample. This training set can be input into the computer device by a user or other device, or it can be constructed by the computer device itself.
[0048] The process of constructing a training set using computer equipment is described below.
[0049] The computer device acquires at least one training sample and a preference score corresponding to each training sample. Each training sample includes a prompt text and a corresponding preference pair. The preference pair includes a first response text and a second response text. The first and second response texts within the same preference pair are two different responses to the same prompt text, which can be understood as different replies to the same prompt text. In other words, each training sample includes a prompt text, a first response text, and a second response text for that prompt text; the first and second response texts constitute a preference pair.
[0050] The preference score corresponding to any training sample is the preference score corresponding to the preference pair within that training sample. The preference score for any preference pair includes a first preference score and a second preference score. The first preference score is the preference score corresponding to the first response text in the preference pair, and the second preference score is the preference score corresponding to the second response text in the preference pair. The preference score corresponding to any response text represents the degree of user preference for that response text. The higher the preference score, the higher the user's preference for that response text; the lower the preference score, the lower the user's preference for that response text. Here, "user" refers to a generalized human. Preference scores can be provided by manual annotation or by a judging model.
[0051] For the first and second response texts in the same preference pair, the first preference score corresponding to the first response text is higher than the second preference score corresponding to the second response text. In other words, for the same prompt text, the user has a higher preference for the first response text. For this prompt text, the user will prioritize replying with the first response text over replying with the second response text.
[0052] The first and second response texts can be manually annotated or generated by different models. Taking the example of different models generating the first and second response texts for the same prompt text, two domain-specific large-scale models with different response quality process the same prompt text separately, obtaining the response texts output by these two large-scale models for that prompt text. The response text output by the domain-specific large-scale model with higher response quality is taken as the first response text, and the response text output by the domain-specific large-scale model with lower response quality is taken as the second response text. The domain-specific large-scale model with higher response quality can be a teacher model, and the domain-specific large-scale model with lower response quality can be a base model.
[0053] For any pair of preferences in a training sample, the computer device obtains the difference score corresponding to the training sample based on the first preference score corresponding to the first response text and the second preference score corresponding to the second response text in the preference pair. For example, the difference score is the difference between the first preference score and the second preference score, and accordingly, the computer device uses the difference between the first preference score and the second preference score as the difference score corresponding to the training sample.
[0054] Assume the training set consists of N training samples, and the nth sample in the N training samples... Each training sample can be represented as , No. The preference pairs in each training sample are .
[0055] in, It is an integer greater than or equal to 1 and less than or equal to N, where N is an integer greater than or equal to 1. For the first The prompt text in each training sample, For the first The first response text in each training sample (i.e., the user's response) (The reply text selected when responding). For the first The second response text from each training sample (i.e., the user's response to...) (The response text was not selected when responding). The corresponding preference score is , The corresponding preference score is , No. The difference scores corresponding to each training sample are: , As shown in the following formula (1).
[0056] (1) Due to preference scores and preference scores It can accurately express the user's opinion and The degree of preference makes the difference score obtained based on these two preference scores... It can accurately represent the difference in user preferences for the two response texts, thereby providing accurate training signals for the model training process, improving training efficiency and the accuracy of the trained domain-specific large model.
[0057] It should be understood that, in response to Relative to the reply text , A high-quality response text is equivalent to , Because they are of poor quality, therefore, in some embodiments, This can also be called a high-quality response text. These can also be referred to as poor-quality response texts. For the first... training samples, difference scores The larger, and The greater the difference in preferences between them, and The more obvious the preference relationship between them, and The greater the difference in quality between them, the better.
[0058] The domain-wide model after SFT is a domain-wide model to be aligned using reinforcement learning. In this embodiment, the domain-wide model after SFT is trained using the improved DPO training process. Each domain-wide model involved in this embodiment can generate a response text based on the input prompt text and output the generated response text.
[0059] Before DPO training, the computer acquires the SFT-enhanced domain model and a copy thereof. The SFT-enhanced domain model is used as the domain model to be trained, and the copy thereof is used as the reference model. For ease of description, the domain model to be trained is referred to as the first domain model, and the reference model is referred to as the second domain model. That is, the first domain model and the second domain model have the same model structure and model parameters, and the second domain model serves as the reference model for the first domain model.
[0060] The computer device performs DPO training on the first domain large model based on the training set and the second domain large model. This DPO training of the first domain large model includes iterative training, which comprises multiple rounds of training. Each round of iterative training includes the following steps 202 and 203.
[0061] In step 202, the computer device obtains the reward value corresponding to the training sample based on the training sample and the domain big model. The domain big model is used to output the response text for the input prompt text.
[0062] Here, the domain-wide model is either the first domain-wide model or the first domain-wide model after at least one round of training. For ease of description, this domain-wide model is referred to as the current domain-wide model, that is, the domain-wide model participating in the training process in the current round. Assuming the current round is the j-th round of iterative training, the current domain-wide model is the first domain-wide model after the (j-1)-th round of training, where j is an integer greater than or equal to 1. When j=1, the current domain-wide model is the first domain-wide model.
[0063] The reward value corresponding to any training sample can be the implicit reward difference for that training sample. The implicit reward difference is a type of reward in the DPO training process.
[0064] Below, taking the domain-wide model participating in the current round of training as the current domain-wide model, the reference model as the second domain-wide model, and the reward value corresponding to the training sample as the implicit reward difference corresponding to the training sample as an example, the method of obtaining the reward value corresponding to the training sample is as follows, combined with steps A1 to A3.
[0065] In step A1, for any training sample, the computer device obtains a first confidence level, a second confidence level, a third confidence level, and a fourth confidence level based on the training sample, the current domain large model, and the second domain large model.
[0066] Among them, the training samples are the first in the training set. 1 sample, the current domain's large model is The second domain's large model is For example, the first confidence level indicates the current domain's large model. For the problem sample Output the first response text The probability, the second confidence level indicates the current domain large model For the problem sample Output the second response text The probability of the third confidence level indicates the second domain of the large model. For the problem sample Output the first response text The probability of the fourth confidence level indicates the size of the two-domain large model. For the problem sample Output the second response text The probability of.
[0067] Using confidence as the basis for a given problem sample in a large domain model Taking the joint log-likelihood of the output response text as an example, the first confidence level is... The second confidence level is The third confidence level is The fourth confidence level is .
[0068] To obtain the first confidence level The second confidence level is For example, let's take the problem sample as an example. Input the current domain's large model Current large-scale model in the field Based on input question samples Output the first intermediate response text, which is the current domain's large model. For the problem sample The generated response text. The computer device is based on the first intermediate response text and the first response text. The difference between them, to obtain the first confidence level Based on the first intermediate response text and the second response text The difference between them is used to obtain the second confidence level. .
[0069] To obtain the third confidence level and the fourth confidence level is For example, let's take the problem sample as an example. Input the second domain large model Second domain large model Based on input question samples Output the second intermediate response text, which is the second domain large model. For the problem sample The generated response text. The computer device is based on the second intermediate response text and the first response text. The difference between them, to obtain the third confidence level Based on the second intermediate response text and the second response text The difference between them is used to obtain the fourth confidence level. .
[0070] In step A2, the computer device determines the implicit reward corresponding to the first response text in the training sample based on the first confidence level and the third confidence level, and determines the implicit reward corresponding to the second response text in the training sample based on the second confidence level and the fourth confidence level.
[0071] The implicit reward for the first response text is the difference between the first and third confidence levels, and the implicit reward for the second response text is the difference between the second and fourth confidence levels.
[0072] With the first Taking one sample as an example, assuming the first confidence level is... The third confidence level is Then the first reply text Corresponding implicit reward As shown in formula (2) below, assuming the second confidence level is The fourth confidence level is Second response text Corresponding implicit reward As shown in formula (3) below.
[0073] (2) (3) For the For each sample, the computer device retrieves the first response text according to formula (2). Corresponding implicit reward According to formula (3), the second response text is obtained. Corresponding implicit reward .
[0074] In step A3, the computer device obtains the reward value corresponding to the training sample based on the implicit reward corresponding to the first response text and the implicit reward corresponding to the second response text.
[0075] The reward value corresponding to the training sample is the implicit reward difference of the training sample, which is the difference between the implicit reward corresponding to the first response text and the implicit reward corresponding to the second response text in the training sample.
[0076] With the first Taking the first sample as an example, the first... Implicit reward difference for each sample As shown in formula (4) below. The computer device is based on the first response text. Corresponding implicit reward Second response text Corresponding implicit reward Calculate the first according to the following formula (4) Implicit reward difference for each sample , will the Implicit reward difference for each sample As the first The reward value corresponding to each sample. For ease of description, the following will use... Indicates the first The reward value corresponding to each sample.
[0077] (4) In step 203, the computer device performs direct preference optimization training on the large model of the domain based on the difference scores and reward values corresponding to the training samples.
[0078] The domain-specific large model is the current domain-specific large model in step 202, and the training performed on the domain-specific large model is DPO training.
[0079] During DPO training, the current domain big model learns the ranking rules corresponding to human preferences based on the relative superiority or inferiority of the response texts within the training samples. By increasing the probability of outputting high-quality response texts and decreasing the probability of outputting low-quality response texts through the output probability distribution of the current domain big model, the deviation between the current domain big model and the reference model (i.e., the second domain big model) is controlled under hyperparameter constraints, so that the response texts generated by the trained domain big model conform to human (i.e., user) preferences.
[0080] In existing DPO training processes, only the best response text (i.e., the first response text) and the worst response text (i.e., the second response text) within the same preference pair of training samples can be identified. The domain model is trained solely based on the implicit reward difference corresponding to the training samples, thus allowing the domain model to learn the knowledge contained in multiple training samples equally (such as the relative superiority or inferiority relationship between response texts within the same preference pair). However, multiple training samples are different, and the knowledge they contain is different. The domain model's equal learning of the knowledge contained in multiple training samples will result in response texts generated by the trained domain model that do not conform to human preferences (i.e., do not meet expectations), leading to poor inference performance.
[0081] The greater the difference in preference between different response texts for the same prompt text in a preference pair, the more worthwhile the domain-wide model involved in training is to learn the superiority-inferiority relationship between the different response texts in the preference pair. Learning this superiority-inferiority relationship can increase the probability that the domain-wide model outputs a high-quality response text that conforms to human preferences for the prompt text, and reduce the probability of outputting a low-quality response text that does not conform to human preferences, thereby improving the inference performance of the domain-wide model.
[0082] In the method provided in this disclosure, the difference score corresponding to the training sample is used to characterize the user's preference differences between different response texts for the same prompt text. Before training the domain big data model, the difference score corresponding to the training sample is obtained first. During the DPO training process of the domain big data model, the reward value and difference score corresponding to the training sample are comprehensively considered, so that the domain big data model can learn the knowledge contained in different training samples in a differentiated manner based on the difference score (such as the relative superiority or inferiority relationship between different response texts for the same text in the same preference pair). For example, if the difference score corresponding to any training sample is larger, the preference difference between different response texts for the same prompt text in that training sample is larger, and the domain big data model learns the knowledge contained in that training sample more effectively. If the difference score corresponding to any training sample is smaller, the preference difference between different response texts for the same prompt text in that training sample is smaller, and the domain big data model learns the knowledge contained in that training sample less effectively. This method of the domain big data model learning the knowledge contained in different training samples in a differentiated manner during the DPO training process can make the response text generated by the trained domain big data model more in line with human preferences, thereby improving the reasoning performance of the trained domain big data model.
[0083] Below, in conjunction with Figure 3 The data processing method provided in the embodiments of this disclosure will be described in detail.
[0084] Figure 3This is a flowchart illustrating another data processing method according to an exemplary embodiment, see [link to flowchart]. Figure 3 This data processing method is applied to a computer device, which can be either the terminal 101 mentioned above or the server 102 mentioned above. For example... Figure 3 As shown, the data processing method includes steps 301 to 304.
[0085] In step 301, the computer device acquires training samples and the corresponding difference scores for the training samples. The training samples include a first response text and a second response text for the same prompt text. The difference scores represent the user's preference difference for the first response text and the second response text.
[0086] Step 301 is the same as step 201 above, and will not be repeated here.
[0087] In step 302, the computer device obtains the reward value corresponding to the training sample based on the training sample and the domain big model. The domain big model is used to output response text for the input prompt text.
[0088] Steps 302 and 202 are similar and will not be repeated here.
[0089] In step 303, the computer device determines the loss value corresponding to the training sample based on the difference score and reward value corresponding to the training sample.
[0090] The loss value corresponding to the training sample is the sample-level loss value.
[0091] The computer device can first normalize the difference scores corresponding to the training samples to a range of 0 to 1, and then determine the loss value corresponding to the training sample based on the normalized difference scores. For example, the process shown in steps B1 and B2 below.
[0092] In step B1, for any training sample, the computer device determines the weight corresponding to the training sample based on the target coefficient value of the smoothing coefficient and the difference score corresponding to the training sample.
[0093] The weights range from 0 to 1, and are normalized difference scores. The weight corresponding to any training sample indicates the reliability of that training sample. The smoothing coefficient controls the smoothness of the weight's change with the difference score; a larger smoothing coefficient means the weight changes faster with the corresponding difference score, and a smaller smoothing coefficient means the weight changes slower. The curve showing the weight's change with the difference score is called the weight change curve, and the smoothing coefficient controls the smoothness of the weight change curve.
[0094] The target coefficient is a value of the smoothing coefficient. Optionally, the target coefficient is a preset fixed value. For example, the range of the smoothing coefficient includes multiple values, and the target coefficient is any value within that range. For instance, the target coefficient is the median of the differences between multiple coefficient values within that range; for example, if the range is 0 to 50, the target coefficient can be 10, or any other value other than 10.
[0095] For any training sample, the computer device scales the difference score corresponding to that training sample based on the target coefficient value of the smoothing coefficient, obtaining the scaled difference score for that training sample. The scaled difference score is the ratio between the target coefficient value and the difference score corresponding to that training sample. (The last sentence appears to be incomplete and requires further context.) Taking the nth training sample as an example, the target coefficient value of the smoothing coefficient is T, and the nth training sample is T. The scaled difference scores corresponding to each training sample are: .
[0096] The computer device uses an activation function to perform a non-linear mapping on the scaled difference score corresponding to the training sample, thereby obtaining the weight corresponding to the training sample. The scaled difference score corresponding to the training sample is the input of the activation function, and the weight corresponding to the training sample is the output of the activation function. The activation function includes, but is not limited to, the Sigmoid function or other functions besides the Sigmoid function.
[0097] With the first Taking a training sample as an example, assuming the activation function is the standard Sigmoid function, the Sigmoid function is expressed as: , , for The input, the first The scaled difference scores corresponding to each training sample are: , No. The weights corresponding to each training sample are: ,but The Sigmoid function can be represented by the following formula (5), where the function curve is the weight change curve.
[0098] (5) For example, computer equipment with the first Scaled difference scores corresponding to each training sample The input of the Sigmoid function According to (5) above, obtain the weights. .
[0099] The above explanation uses the scaled difference scores corresponding to the training samples as the input to the activation function. In other embodiments, an activation function with a smoothing coefficient is used to process the... The difference scores corresponding to each training sample are used to obtain the weights. The activation function with a smoothing coefficient is called the target activation function. Target activation functions include, but are not limited to, the sigmoid function or other functions besides the sigmoid function. The input to the target activation function is the nth... The difference scores corresponding to each training sample are represented by the function curve of the target activation function, which is the weight change curve.
[0100] In the target activation function, the smoothing coefficient is used to smooth the first... The difference scores corresponding to each training sample Scaling is applied, and the smoothing coefficient is set to the target coefficient value. The output of the target activation function is the first... For example, assuming the target activation function is a training sample. , The input to the target activation function is given by the following formula (6).
[0101] (6) For the The training sample, the computer device with the first training sample The difference scores corresponding to each training sample Target activation function Input According to the above formula (6), the weights are obtained. That is to say, As Input target activation function , target activation function The output is used as weights .
[0102] The smoothing coefficient can control the trend of the activation function curve. Taking the Sigmoid function as an example, when the target smoothing coefficient value is 1, the curve of the Sigmoid function is relatively steep; when the target smoothing coefficient value is 10, the curve of the Sigmoid function is smooth; and when the target smoothing coefficient value is 50, the curve of the Sigmoid function is gentle. The smooth Sigmoid function is suitable for general scenarios. Therefore, by reasonably selecting the target sensitivity coefficient value, we can select weights that are beneficial to the actual training scenario and improve the inference effect of large-scale domain models.
[0103] Based on the coefficient values of the smoothing coefficient and the activation function, the difference scores corresponding to the training samples can be reasonably normalized to ensure the accuracy of the weights of the obtained training samples. This can provide accurate learning preference strength for model training, making it easier for the domain-wide model participating in the training to learn the knowledge contained in the training with appropriate strength, thereby improving the inference performance of the trained domain-wide model.
[0104] It should be understood that the first The difference score corresponding to each training sample The larger, the more The weights corresponding to each training sample The larger, the more The more reliable the training samples, the better. The more knowledge contained in a training sample, the more it should be learned by the domain-specific large model. The smaller, the higher the weight The smaller, the first The less reliable the training sample, the more reliable the training sample becomes. Training samples may contain noise, and the knowledge contained in noisy training samples should be learned in a limited way. During training, if the current domain-wide model can focus on learning the knowledge contained in reliable samples and lightly learn the knowledge contained in unreliable samples, it can increase the probability of the trained domain-wide model outputting high-quality response text to the prompt text, thereby improving the inference performance of the domain-wide model.
[0105] Based on this, the embodiments of this disclosure use the difference scores corresponding to the training samples as prior signals of the preference learning intensity of the domain large model, and inject the difference scores into the training target so that during the training process, the current domain large model focuses on learning the knowledge contained in reliable samples and lightly learns the knowledge contained in unreliable samples, thereby increasing the probability of the trained domain large model outputting high-quality response text for prompt text, and thus improving the reasoning effect of the trained domain large model.
[0106] The training objective is to minimize the loss value corresponding to the training samples. The difference score can be injected into the training objective through the following step B2.
[0107] In step B2, the computer device determines the loss value corresponding to the training sample based on the weight and reward value corresponding to the training sample.
[0108] Wherein, the loss value corresponding to any training sample is the product of the weight corresponding to the training sample and the reward value corresponding to the training sample. Wherein, the loss value corresponding to the training sample is the loss value calculated based on the loss function of a single training sample, and the weight and reward value corresponding to the training sample are the inputs of the loss function of a single training sample, and the loss function of a single training sample is shown in the following formula (7).
[0109] With the first Taking the training sample as an example, the computer device determines the training sample according to the following formula (7). The loss value corresponding to the training sample. Combining the above formulas (4) and (6), formula (7) can be transformed into the following formula (8). The computer device can also determine the loss value of the training sample according to the following formula (8). The loss value corresponding to each training sample.
[0110] (7) (8) In this context, both formulas (7) and (8) are sample-level loss functions. For the first The loss value corresponding to each training sample; The standard Sigmoid function maps the input numerical value to [0,1]. for Input, for hyperparameters, Used to control the loss function Compared to The weight, It can be configured according to the actual application scenario. For the first The reward value corresponding to each training sample For the first The weights corresponding to each training sample.
[0111] The above description uses a fixed target coefficient value for the smoothing coefficient as an example. In some embodiments, the target coefficient value is a dynamic value, changing with the difference score corresponding to the training sample. For example, before determining the weight corresponding to any training sample, the computer device obtains the target coefficient value from multiple candidate coefficient values of the smoothing coefficient based on the difference score and reward value corresponding to that training sample.
[0112] Here, candidate coefficients are the coefficient values that are candidates for the smoothing coefficient. For example, multiple candidate coefficient values include all coefficient values within the range of the smoothing coefficient. Target coefficients are the candidate coefficient values that minimize the loss value corresponding to the training sample.
[0113] For example, for the first For each training sample, the computer device employs a target optimization strategy, using the loss value corresponding to the training sample. The minimum value is taken as the optimization objective, based on the first... The difference scores corresponding to each training sample and reward value From the range of smoothing coefficient values, a search is conducted to find coefficient values that satisfy the optimization objective. These found coefficient values that satisfy the optimization objective are then determined as the target coefficient values. This optimization strategy includes, but is not limited to, grid search algorithms.
[0114] The target coefficient value of the smoothing coefficient changes with the difference score corresponding to the training sample, allowing the weights of different training samples with different difference scores to follow the changes in the training sample differentially. This increases the accuracy of the weights corresponding to individual training samples. Injecting accurate weights as a prior signal for the preference learning intensity into the training objective enables the current domain-wide model to precisely control the learning intensity of the training samples, further improving the inference performance of the trained domain-wide model. With the goal of minimizing the loss value of the training samples, the target coefficient value is obtained from multiple candidate coefficient values of the smoothing coefficient. The obtained target parameter is used to calculate the weights, and then the loss value of the training samples is calculated based on the calculated weights, so that the loss value of the training samples can converge quickly, improving training efficiency.
[0115] In step 304, the computer device performs DPO training on the large model of the domain based on the loss value corresponding to the training sample.
[0116] Among them, the large model in this domain refers to the large model in the current domain that participates in the training.
[0117] When the training set includes a single training sample, the computer device performs DPO training on a large model in the domain with the optimization objective of the loss value corresponding to that training sample converging to the target value range.
[0118] When the training set includes multiple training samples, the computer device obtains the difference score corresponding to each training sample in the same way as obtaining the difference score corresponding to a single training sample, obtains the weight corresponding to each training sample in the same way as obtaining the weight corresponding to a single training sample, and obtains the loss value corresponding to each training sample in the same way as obtaining the loss value corresponding to a single training sample, thereby ensuring that each training sample corresponds to a difference score, a loss value, and a weight. In this case, step 304 can be implemented through the processes shown in steps C1 and C2 below.
[0119] In step C1, the computer device obtains the total loss value based on the loss values and weights corresponding to multiple training samples.
[0120] The total loss value is the total loss value of the multiple training samples in this round of training, which is used to characterize the degree of preference ranking error of the current domain large model on the preference pairs of the multiple training samples.
[0121] The computer device calculates a weighted average of the loss values corresponding to the multiple training samples based on their respective weights, thus obtaining the total loss value.
[0122] For example, suppose that this round of training involves multiple training samples of N, and the Nth training sample is... The weights corresponding to each training sample are: , No. The loss value corresponding to each training sample is The computer equipment obtains the total loss value according to the following formula (9). .
[0123] (9) Wherein, formula (9) is the loss function in the DPO training process, which is the objective function used to optimize the model parameters of the domain model.
[0124] By obtaining the total loss value during this training round according to weights and weighted averages, the total loss value can more accurately and comprehensively represent the ability of the current domain-wide model to output high-quality response text to prompt text. Based on this total loss value, the model parameters of the current domain-wide model are updated, which can make the total loss value converge as soon as possible, thereby improving training efficiency.
[0125] In step C2, the computer device updates the model parameters of the domain large model based on the total loss value.
[0126] Among them, the large model in this domain refers to the current domain-specific large model that participated in the training. .
[0127] Computer devices utilize current domain-wide models The total loss value L obtained in this round of training (i.e., the total loss value obtained in step C1) is backpropagated to calculate the total loss value L relative to the current domain large model. The gradient of the model parameters; based on this gradient, following the gradient descent principle, the large model in the current domain is analyzed. The model parameters are updated to obtain the updated domain-wide model. This updated domain-wide model is then used as the current domain-wide model for the next round of training. Based on multiple training samples in the training set, this study analyzes large-scale models in the current domain. One round of training is performed. This process is repeated for multiple rounds, with the total loss value L continuously decreasing until it converges to a value within the target range. The final domain-wide model with updated parameters is the trained domain-wide model (referred to as the third domain-wide model). This achieves the transformation of the first domain-wide model into a third domain-wide model. For the same prompt text, the response text output by the third domain-wide model is more in line with user preferences compared to the first domain-wide model. During this training process, the model parameters of the second domain-wide model remain unchanged, and the second domain-wide model is always used as a reference model.
[0128] The process described in steps C1 and C2 above, when obtaining the total loss value of this round of training, considers not only the loss value corresponding to each training sample, but also the weight corresponding to each training sample. Different training samples have different weights, which will make different training samples contribute differently to the total loss value. This allows the total loss value of this round of training to accurately and comprehensively represent the ability of the current domain-wide model to output high-quality response text to the prompt text. Based on this total loss value, the model parameters of the current domain-wide model are updated, which can make the total loss value reach a convergence state as soon as possible, thereby improving training efficiency.
[0129] It should be understood that in any round of training, the difference score corresponding to the training sample is used as the prior signal of the preference learning intensity of the domain large model. The weight corresponding to the training sample is determined based on the difference score corresponding to the training sample, and then the total loss value of this round of training is calculated based on the weight corresponding to the training sample. This realizes the injection of the prior signal of preference learning intensity into the training target. For each training sample used in this round of training, if the difference score of a training sample is larger, the weight of that training sample is larger, and its contribution to the total loss value in this round of training is greater. Updating the model parameters of the current domain-wide model based on the total loss value enables the current domain-wide model to learn the knowledge contained in that training sample with a higher preference learning intensity. Conversely, if the difference score of a training sample is smaller, it indicates that the training sample contains noise. The smaller the difference score of the training sample, the greater the noise in the training sample, the smaller the weight of that training sample, and the smaller its contribution to the total loss value in this round of training. Updating the model parameters of the current domain-wide model based on the total loss value enables the current domain-wide model to learn the knowledge contained in that training sample with a relatively small preference learning intensity, thus achieving lightweight learning. This achieves soft screening and automatic noise reduction of training samples during the training process.
[0130] The processes described in steps 303 and 304 above are one possible implementation of step 203 above. In this embodiment of the disclosure, the difference score serves as a priori signal of the preference learning intensity of the domain-wide model, and minimizing the loss value corresponding to the training samples is the objective of model training.
[0131] The following is based on Figure 4 Taking one example, we will provide an exemplary description of any round of training in the above DPO training process.
[0132] like Figure 4 As shown, the training set includes multiple training samples, each of which includes a prompt text. A preference pair and difference score Preferences include those for prompt text. The two response texts contain response prompt text. The user's preferred response text is The text of the reply when the user does not select is Response text and reply text The difference between the preference scores of the two response texts is the difference score, which indicates that the two texts have different preference scores. .
[0133] During any round of training of the domain-wide model, for any training sample, the computer device will display the prompt text in that training sample. Input the domain-wide model and the reference model (not shown in the figure) respectively, and obtain the domain-wide model for the prompt text. The response text and the reference model for the prompt text The response text, the computer device, is based on the domain-specific model for the prompt text. The response text and the reference model for the prompt text The implicit reward value corresponding to the training sample is calculated based on the response text and the preference pair in the training sample. The implicit reward value corresponding to the training sample is then used in conjunction with the difference score in the training sample. The reward value corresponding to the training sample (i.e., the sample-level reward value) is calculated. Similarly, the reward value corresponding to each training sample in the training set can be calculated. Based on the reward values corresponding to each training sample, the total loss value for this round of training is calculated, thus injecting the difference score into the training objective. The computer device updates the model parameters of the domain-wide model according to the total loss value of this round of training. This round of training ends. If the DPO training termination condition is not met, the domain-wide model with updated model parameters is trained again in the same way until the DPO training termination condition is met, at which point the DPO training ends, and the final domain-wide model obtained is the trained domain-wide model.
[0134] In the method provided in this disclosure embodiment, when determining the loss value corresponding to the training sample, not only the reward value corresponding to the training sample is considered, but also the difference score corresponding to the training sample is considered. This allows the prior signal of the preference learning intensity to be injected into the training target, enabling the domain big model n participating in the training process to focus on learning the knowledge contained in the training samples with larger difference scores and lightly learn the knowledge contained in the training samples with smaller difference scores. This achieves differentiated learning of the knowledge contained in different training samples, making the response text generated by the domain big model more in line with human preferences, thereby improving the reasoning performance of the trained domain big model.
[0135] Compared to the standard DPO training process, the improved DPO training process described above results in more stable loss changes during reinforcement learning training of large domain models. For example... Figure 5 As shown, loss change curve 51 is the curve showing the change of loss value (e.g., total loss value) with the number of training steps (i.e., training rounds) during the training of a large domain model using the standard DPO training process. Loss change curve 52 is the curve showing the change of loss value (e.g., total loss value) with the number of training steps during the training of a large domain model using the improved DPO training process of this disclosure. Compared with loss change curve 51, loss change curve 52 is smoother and has no obvious oscillations. From loss change curve 52, it can be seen that during the training of a large domain model using the standard DPO training process, the loss value can smoothly decrease from the initial 0.35 to 0.05~0.08, with stable convergence and no obvious oscillations. It can converge when the number of training steps (i.e., training rounds) reaches 100, which can reflect the characteristics of efficient learning.
[0136] Compared to the standard DPO training process, the improved DPO training process described above results in more stable accuracy variations when training large domain models using reinforcement learning. For example... Figure 6 As shown, accuracy curve 61 represents the change in accuracy of the domain-wide model with training steps during training using the standard DPO training process, while accuracy curve 62 represents the change in accuracy of the domain-wide model with training steps during training using the improved DPO training process of this disclosure. Accuracy curve 62 shows that during training the domain-wide model using the improved DPO training process of this disclosure, the accuracy of the domain-wide model can rapidly increase from 0.34 to 0.95-0.98, reaching the industry-ideal range of 0.9-1.0. The accuracy of the domain-wide model can ultimately stabilize at 0.96, indicating that the trained domain-wide model achieves excellent performance. This ideal state signifies excellent model performance, efficient learning during the reinforcement learning phase, and fast and stable convergence of the trained model.
[0137] To verify the effectiveness of the improved DPO training process provided in this disclosure, training was conducted in both vision and language (Vision-Language) training methods. Multiple sets of comparative experiments were conducted in multimodal scenarios including Language (VL) and LLM plain text scenarios. Multiple baseline models, a supervised fine-tuning (SFT) experimental group, and a reinforcement learning (RL) preference alignment experimental group were set up for evaluation and comparison. The experimental results are as follows: Figure 7 and Figure 8 As shown.
[0138] Figure 7 To compare the evaluation results in a VL multimodal scenario, the test task was image-to-text generation (i.e., visual-language). Two external baselines were set up: the original baseline model and the distillation control model. The original baseline model scored 78.91, while the distillation control model scored 92.13. The experiment was divided into an SFT phase and a RL phase. In the SFT phase, a Mixture of Experts (MoE)-based SFT (i.e., MoE) model was used. The SFT (Solved Frame Theory) was used to jointly train the VL (Domain-Specific) multimodal large model (i.e., the domain-specific large model) to obtain the SFT-adjusted VL multimodal large model. The evaluation score of the SFT-adjusted VL multimodal large model was 81.69, an improvement of 2.78 compared to the original baseline. In the RL (Reinforcement Learning) phase, the standard DPO (Domain-Specific Programming) training procedure was used to train the SFT-adjusted VL multimodal large model to obtain the RL-adjusted VL multimodal large model. The evaluation score of the RL-adjusted VL multimodal large model was 84.60, an improvement of 2.91 compared to the original baseline. The SW-DPO (Solved Frame Theory) training procedure was used to train the SFT-adjusted VL multimodal large model to obtain the RL-adjusted VL multimodal large model. The evaluation score of the RL-adjusted VL multimodal large model was 88.86, an improvement of 4.26 compared to the original baseline. The evaluation results show that supervised fine-tuning alone has limited effect on improving model performance, while introducing a preference alignment mechanism can further improve model performance. Compared to the standard DPO training process, the improved DPO provided in this disclosure achieves the best evaluation results, significantly outperforming SFT and the standard DPO, and approaching the performance of DeepSeek. V4 The Pro control model was used to verify the effectiveness of the improved scheme in multimodal scenarios.
[0139] Figure 8To compare the evaluation results in the LLM scenario, the test task was to generate the golden formula from storyboard information. The baseline model scored 92.00; the native untuned pedestal model scored 80.40; and the pure Mixture of Experts (MoE) pedestal model scored 74.24. In the SFT stage, a low-rank adaptive (Low) formula was used. Rank Adaptation (LoRA) fine-tuned SFT was applied to the LLM large model (i.e., the domain large model) to obtain an SFT-adjusted LLM large model with an evaluation score of 90.10. Data augmentation SFT using high-quality data was applied to the LLM large model to obtain an SFT-adjusted LLM large model with an evaluation score of 90.50. Improved SFT using format repair was applied to the LLM large model to obtain an SFT-adjusted LLM large model with an evaluation score of 78.30. Data augmentation SFT was applied to the LLM large model to obtain an SFT-adjusted LLM large model with an evaluation score of 91.10. In the RL stage, MoE SFT joint training was used to train the SFT-adjusted LLM large model, resulting in an RL-adjusted LLM large model with an evaluation score of 90.02. SW was used... The DPO training process involves performing RL training on the same SFT-based LLM large model to obtain an RL-based LLM large model. The evaluation score of this RL-based LLM large model is 90.10, which is 15.86 higher than that of the pure MoE base and the evaluation score is close to the control baseline. At the same time, this approach reduces the inference time from 19s to 6.5s, improving the inference speed by 65%. It significantly reduces the inference time while maintaining high evaluation accuracy, thus balancing model performance and inference efficiency.
[0140] Therefore, based on observations from real-world business scenarios, SW DPO also outperforms other methods and ablation experiments in terms of accuracy, as mentioned above. Figure 7 and Figure 8 As shown, compared with the base model, the training model distilled from the higher-order model, SW DPO reduced inference costs by 60%, but the performance did not decrease significantly. Therefore, SW DPO can accelerate inference and has good engineering application value.
[0141] It should be understood that the aforementioned domain-wide model is a question-answering model. A question-answering model has the function of outputting response text in response to input prompt text. The input of the question-answering model is the prompt text, and the output is the response text to the prompt text. The domain-wide model is one example of a question-answering model. In some embodiments, the question-answering model may also have other types of models besides the domain-wide model. The above data processing method is applicable to any question-answering model; that is, a computer device can train any question-answering model using the above data processing method. The question-answering model can be any language model among LM, LLM, or multimodal language models.
[0142] Figure 9 This is a logical structure block diagram of a data processing apparatus according to an exemplary embodiment, with reference to... Figure 9 The data processing device includes: The first acquisition unit 901 is configured to acquire training samples and the difference scores corresponding to the training samples. The training samples include a first response text and a second response text for the same prompt text. The difference scores represent the user's preference difference for the first response text and the second response text. The second acquisition unit 902 is configured to perform an operation based on the training samples and the domain big model to acquire the reward value corresponding to the training samples, wherein the domain big model is used to output response text for the input prompt text. Training unit 903 is configured to perform direct preference optimization training on the domain large model based on the difference scores and reward values corresponding to the training samples.
[0143] Optionally, the training unit 903 includes: The subunit is configured to perform the determination of the loss value corresponding to the training sample based on the difference score and reward value corresponding to the training sample. The training subunit is configured to perform direct preference optimization training on the domain-wide model based on the loss values corresponding to the training samples.
[0144] Optionally, the determining subunit includes: The weight determination subunit is configured to perform a target coefficient value based on a smoothing coefficient and the difference score to determine the weights corresponding to the training samples, wherein the weights indicate the reliability of the training samples, and the smoothing coefficient is used to control the smoothness of the weights as the difference score changes. The loss determination subunit is configured to determine the loss value corresponding to the training sample based on the weight and reward value corresponding to the training sample.
[0145] Optionally, the weight determination subunit is configured to perform: Based on the target coefficient value, the difference score is scaled to obtain the scaled difference score; The weights are obtained by non-linearly mapping the scaled difference scores using an activation function.
[0146] Optionally, the target coefficient value varies with the difference score.
[0147] Optionally, the data processing apparatus further includes: The third acquisition unit is configured to perform an operation based on the difference score and reward value corresponding to the training sample to acquire the target coefficient value from a plurality of candidate coefficient values of the smoothing coefficient, wherein the target coefficient value is a candidate coefficient value that minimizes the loss value.
[0148] Optionally, there are multiple training samples, and each training sample corresponds to a loss value and a weight; The training subunit includes: The sub-unit is configured to perform a total loss value based on the loss values and weights corresponding to multiple training samples. The update subunit is configured to update the model parameters of the domain large model based on the total loss value.
[0149] Optionally, the sub-unit is configured to perform a weighted average of the loss values corresponding to the plurality of training samples based on the weights corresponding to the plurality of training samples to obtain the total loss value.
[0150] Optionally, the first acquisition unit 901 is configured to acquire the difference score based on the preference score corresponding to the first response text and the preference score corresponding to the second response text, wherein the preference score represents the degree of preference of the user for the corresponding response text.
[0151] Regarding the data processing apparatus in the above embodiments, the specific manner in which each unit performs its operations has been described in detail in the embodiments related to the data processing method, and will not be elaborated upon here.
[0152] Figure 10 This is a schematic diagram illustrating the structure of a computer device according to an exemplary embodiment. Figure 10The computer device 1000 shown may vary considerably due to differences in configuration or performance. It may include one or more Central Processing Units (CPUs) 1001 and one or more memories 1002. The memory 1002 stores at least one line of program code, which is loaded and executed by the processor 1001 to implement the data processing methods provided in the various embodiments described above. Of course, the computer device 1000 may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The computer device 1000 may also include other components for implementing device functions, which will not be elaborated here.
[0153] In an exemplary embodiment, a computer-readable storage medium including at least one instruction is also provided, such as a memory including at least one instruction, which can be executed by a processor in a computer device to complete the data processing method in the above embodiments. Optionally, the computer-readable storage medium may be a non-transitory computer-readable storage medium, such as ROM (Read-Only Memory), RAM (Random-Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk, and optical data storage device, etc.
[0154] In an exemplary embodiment, a computer program product is also provided, including one or more instructions that can be executed by a processor of a computer device to perform the data processing methods provided in the various embodiments described above.
[0155] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.
[0156] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims of this disclosure.
[0157] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.
Claims
1. A data processing method, characterized in that, include: Obtain training samples and the corresponding difference scores for the training samples. The training samples include a first response text and a second response text for the same prompt text. The difference scores represent the user's preference difference for the first response text and the second response text. Based on the training samples and the domain big model, the reward value corresponding to the training samples is obtained, and the domain big model is used to output the response text for the input prompt text; Based on the difference scores and reward values corresponding to the training samples, the domain-wide model is trained using direct preference optimization.
2. The data processing method according to claim 1, characterized in that, The step of performing direct preference optimization training on the domain-wide model based on the difference scores and reward values corresponding to the training samples includes: Based on the difference score and reward value corresponding to the training sample, determine the loss value corresponding to the training sample; Based on the loss values corresponding to the training samples, the domain-wide model is trained using direct preference optimization.
3. The data processing method according to claim 2, characterized in that, The step of determining the loss value corresponding to the training sample based on the difference score and reward value corresponding to the training sample includes: Based on the target coefficient value of the smoothing coefficient and the difference score, the weights corresponding to the training samples are determined. The weights indicate the reliability of the training samples, and the smoothing coefficient is used to control the smoothness of the weights as the difference score changes. Based on the weights and reward values corresponding to the training samples, the loss value corresponding to the training samples is determined.
4. The data processing method according to claim 3, characterized in that, The determination of the weights corresponding to the training samples based on the target coefficient value of the smoothing coefficient and the difference score includes: Based on the target coefficient value, the difference score is scaled to obtain the scaled difference score; The weights are obtained by non-linearly mapping the scaled difference scores using an activation function.
5. The data processing method according to claim 3, characterized in that, The target coefficient value changes as the difference score changes.
6. The data processing method according to claim 5, characterized in that, The method further includes: Based on the difference score and reward value corresponding to the training sample, the target coefficient value is obtained from multiple candidate coefficient values of the smoothing coefficient. The target coefficient value is the candidate coefficient value that minimizes the loss value.
7. The data processing method according to claim 3, characterized in that, There are multiple training samples, and each training sample corresponds to a loss value and a weight. The step of performing direct preference optimization training on the domain-wide model based on the loss values corresponding to the training samples includes: The total loss value is obtained based on the loss values and weights corresponding to multiple training samples; Based on the total loss value, the model parameters of the large domain model are updated.
8. The data processing method according to claim 7, characterized in that, The process of obtaining the total loss value based on the loss values and weights corresponding to multiple training samples includes: Based on the weights corresponding to the multiple training samples, the loss values corresponding to the multiple training samples are weighted and averaged to obtain the total loss value.
9. The data processing method according to any one of claims 1-8, characterized in that, The process of obtaining the difference score includes: Based on the preference score corresponding to the first response text and the preference score corresponding to the second response text, the difference score is obtained, and the preference score represents the degree of preference of the user for the corresponding response text.
10. A data processing apparatus, characterized in that, include: The first acquisition unit is configured to acquire training samples and the difference scores corresponding to the training samples. The training samples include a first response text and a second response text for the same prompt text. The difference scores represent the user's preference difference for the first response text and the second response text. The second acquisition unit is configured to perform an operation based on the training samples and the domain big model to acquire the reward value corresponding to the training samples, wherein the domain big model is used to output response text for the input prompt text. The training unit is configured to perform direct preference optimization training on the domain large model based on the difference scores and reward values corresponding to the training samples.
11. A computer device, characterized in that, include: One or more processors; One or more memories for storing the one or more processor-executable instructions; The one or more processors are configured to execute the instructions to implement the data processing method as described in any one of claims 1 to 9.
12. A computer-readable storage medium, characterized in that, When at least one instruction in the computer-readable storage medium is executed by one or more processors of a computer device, the computer device is enabled to perform the data processing method as described in any one of claims 1 to 9.
13. A computer program product, characterized in that, It includes one or more instructions, which are executed by one or more processors of a computer device, enabling the computer device to perform the data processing method as described in any one of claims 1 to 9.
