Training methods for reward models, optimization methods for large language models, and related equipment.

Abstract

This invention discloses a training method for a reward model, an optimization method for a large language model, and related methods. The training method for the reward model includes: obtaining preference training sample pairs and a reward model to be trained, wherein the preference training sample pairs include preferred response samples and non-preferred response samples; calculating the reward score difference between the preferred response samples and non-preferred response samples based on the reward model to be trained; constructing a cost matrix based on the reward score difference and the semantic correlation between the preferred response samples and non-preferred response samples; calculating the loss margin based on the cost matrix; calculating the pairwise preference loss value with the margin based on the loss margin, and updating the parameters of the reward model to be trained with the goal of minimizing the loss value with the margin, thereby obtaining a trained reward model. This improves the model's learning ability on difficult samples and its overall generalization performance, avoids over-reliance on simple samples, and thus improves the generation quality of large language models in complex tasks.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to a reward model training method, a large language model optimization method and related equipment. BACKGROUND

[0002] Large language models (LLM) exhibit strong generation capabilities in natural language processing tasks, but their generation quality often relies on reinforcement learning from human feedback (RLHF) processes for optimization. Among them, the training of reward models is a key link of RLHF, and common methods are mostly based on the Bradley-Terry (BT) model idea, which learns through the comparison of paired preference samples, so that the predicted score of the preferred response is higher than that of the non-preferred response.

[0003] However, the existing BT-based reward model training method has the following problems: first, the BT model mainly focuses on the high-low relationship between the preferred response and the non-preferred response, and ignores the relative size of the score difference between the two, resulting in insufficient separation degree between reward distributions. Especially in the "difficult sample" scenario where the semantics are similar but the reward difference is subtle, the model is difficult to effectively capture the potential difference, so as to form a clear distinguishing boundary. Second, the model often relies too much on "easy samples" in the training process, and pays insufficient attention to difficult samples, which can easily cause overfitting, thereby weakening the generalization ability in complex scenarios or out-of-distribution samples.

[0004] Therefore, it is necessary to propose a new reward model training method to improve the model's ability to distinguish difficult samples and overall generalization performance, and thus improve the generation quality of large language models in generation tasks. SUMMARY

[0005] Therefore, it is necessary to propose a new reward model training method to improve the model's ability to distinguish difficult samples and overall generalization performance, and thus improve the generation quality of large language models in generation tasks.

[0006] A reward model training method, the method comprising:

[0007] Obtaining a pair of preference training samples and a reward model to be trained, the pair of preference training samples comprising a preferred response sample and a non-preferred response sample;

[0008] Calculating the reward score difference between the preferred response sample and the non-preferred response sample based on the reward model to be trained;

[0009] A cost matrix is ​​constructed based on the difference in reward scores and the semantic correlation between the preferred response sample and the non-preferred response sample;

[0010] Based on the cost matrix, calculate the loss margin;

[0011] The pairwise preference loss value with margin is calculated based on the loss margin, and the parameters of the reward model to be trained are updated with the optimization objective of minimizing the loss value with margin, so as to obtain the trained reward model.

[0012] Optionally, constructing the cost matrix based on the reward score difference and the semantic correlation between the preferred response sample and the non-preferred response sample includes:

[0013] The reward score difference and the semantic relevance are fused according to a preset combination method to obtain a cost value, which is used to characterize the difference between the preferred response sample and the non-preferred response sample;

[0014] Based on the cost values, the cost matrix is ​​constructed.

[0015] Optionally, calculating the loss margin based on the cost matrix includes:

[0016] For each preferred response sample, determine the cost value in the cost matrix corresponding to the preferred response sample as the row and multiple non-preferred response samples as the columns;

[0017] The cost values ​​are aggregated according to preset weights to obtain the loss margin corresponding to the preferred response sample.

[0018] Optionally, calculating the loss margin based on the cost matrix includes:

[0019] The optimal transmission, which characterizes the matching relationship between preferred and non-preferred response samples, is obtained based on the cost matrix.

[0020] For each preferred response sample, a weighted aggregation is performed based on the optimal transmission and the cost value in the cost matrix to obtain the loss margin corresponding to the preferred response sample.

[0021] Optionally, the step of obtaining the optimal transfer based on the cost matrix to characterize the matching relationship between preferred and non-preferred response samples includes:

[0022] Construct a first distribution of the preferred response samples and a second distribution of the non-preferred response samples;

[0023] Determine a feasible region based on the first distribution and the second distribution, wherein the feasible region is used to constrain the row edges of the transfer matrix to be equal to the first distribution and the column edges to be equal to the second distribution;

[0024] Within the feasible region, an optimization problem is established with the inner product of the cost matrix and the transmission matrix as the cost term and including an entropy regularization term;

[0025] The optimal transmission is obtained by solving the optimization problem.

[0026] Optionally, the step of calculating the pairwise preference loss value with a margin based on the loss margin includes:

[0027] For each pair of preferred response samples and non-preferred response samples, calculate the difference in reward scores after adjustment by the loss margin;

[0028] The corrected reward score difference is input into a monotonically increasing probability mapping function to obtain the preference probability of the sample pair;

[0029] The loss value in the form of negative log-likelihood is calculated based on the preference probability, and the loss values ​​of multiple preference training sample pairs are aggregated to obtain the pairwise preference loss value with margin.

[0030] An optimization method for a large language model, the method comprising:

[0031] Obtain a trained reward model and a large language model to be optimized, wherein the trained reward model is obtained by the training method of the reward model;

[0032] The output of the large language model to be optimized is evaluated based on the trained reward model, and a corresponding reward signal is generated.

[0033] The reward signal is used to guide the policy optimization of the large language model to obtain an optimized large language model.

[0034] A training apparatus for a reward model, the apparatus comprising:

[0035] The first acquisition module is used to acquire preference training sample pairs and a reward model to be trained, wherein the preference training sample pairs include preferred response samples and non-preferred response samples;

[0036] The first calculation module is used to calculate the difference in reward scores between the preferred response sample and the non-preferred response sample based on the reward model to be trained.

[0037] The first construction module is used to construct a cost matrix based on the reward score difference and the semantic correlation between the preferred response sample and the non-preferred response sample.

[0038] The second calculation module is used to calculate the loss margin based on the cost matrix;

[0039] The first update module is used to calculate the pairwise preference loss value with margin based on the loss margin, and update the parameters of the reward model to be trained with the optimization objective of minimizing the loss value with margin, so as to obtain the trained reward model.

[0040] An optimization apparatus for a large language model, the apparatus comprising:

[0041] The second acquisition module is used to acquire the trained reward model and the large language model to be optimized. The trained reward model is obtained by the training method of the reward model.

[0042] The first evaluation module is used to evaluate the output of the large language model to be optimized based on the trained reward model and generate a corresponding reward signal.

[0043] The first optimization module is used to use the reward signal to guide the policy optimization of the large language model, so as to obtain an optimized large language model.

[0044] A computer device includes a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor, when executing the computer-readable instructions, implements the training method of the reward model or the optimization method of the large language model described above.

[0045] A readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the training method of the above-described reward model or the optimization method of a large language model.

[0046] The training method for the aforementioned reward model involves obtaining preference training sample pairs and a reward model to be trained. The preference training sample pairs include preferred response samples and non-preferred response samples. The reward score difference between the preferred response samples and non-preferred response samples is calculated based on the reward model to be trained. A cost matrix is ​​constructed based on the reward score difference and the semantic correlation between the preferred and non-preferred response samples. A loss margin is calculated based on the cost matrix. A pairwise preference loss value with a margin is calculated based on the loss margin, and the parameters of the reward model to be trained are updated with the optimization objective of minimizing the loss value with a margin, resulting in a trained reward model. By constructing a cost matrix based on the reward score difference and semantic correlation during training, and calculating the loss margin from it, and then using the loss margin to correct the loss value, the reward model can adaptively adjust the discrimination requirements according to the sample difficulty. This results in a clearer discrimination effect in scenarios with semantically similar but subtly different samples, improving the model's learning ability for difficult samples and its overall generalization performance, avoiding over-reliance on simple samples, and thus improving the generation quality of large language models in complex tasks. Attached Figure Description

[0047] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a flowchart illustrating the training method of a reward model in one embodiment of the present invention;

[0049] Figure 2 This is another flowchart illustrating the training method of the reward model in one embodiment of the present invention;

[0050] Figure 3 This is a flowchart illustrating a method for optimizing a large language model in one embodiment of the present invention;

[0051] Figure 4 This is a schematic diagram of the structure of a training device for a reward model in one embodiment of the present invention;

[0052] Figure 5 This is a schematic diagram of the structure of an optimization device for a large language model in one embodiment of the present invention;

[0053] Figure 6 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] In one embodiment, such as Figure 1 As shown, a method for training a reward model is provided, including the following steps:

[0056] 101. Obtain the preference training sample pairs and the reward model to be trained.

[0057] In this embodiment of the invention, the preference training sample pair includes a preferred response sample and a non-preferred response sample. Specifically, the preference training sample pair can originate from manually labeled preference datasets, comparison data collected by crowdsourcing platforms, or response ranking data obtained through expert evaluation. For example, among multiple candidate responses given the same input prompt, a human or expert selects the response that best meets the requirements as the preferred response sample and selects the relatively poorer response as the non-preferred response sample, thereby forming a preference training sample pair.

[0058] The preferred response sample can be a text response that is semantically complete, logically coherent, and conforms to the facts or task requirements; the non-preferred response sample can be a text response that is semantically incomplete, logically incoherent, contains factual errors, or does not meet the task requirements.

[0059] The reward model to be trained can be a scoring model built on a neural network structure, such as a reward estimation network based on a pre-trained language model, or a model using lightweight parameter tuning methods (such as fine-tuning or parameter-efficient fine-tuning) to generate corresponding reward scores for the input response text.

[0060] In one possible embodiment, the aforementioned preference training sample pairs may further include user prompts, specifically in the form of data such as... ,in To provide users with a prompt, As the preferred response, This is not the preferred response.

[0061] 102. Calculate the difference in reward scores between preferred response samples and non-preferred response samples based on the reward model to be trained.

[0062] In this embodiment of the invention, the reward model to be trained is used to score the input response sample to characterize the overall quality of the response sample in terms of language fluency, semantic reasonableness, factual accuracy, and preference level. The reward score can be a real number, with a larger value indicating that the response is more in line with expectations.

[0063] Specifically, the preferred response sample is input into the reward model to be trained to obtain the reward score of the preferred response sample; the non-preferred response sample is input into the reward model to be trained to obtain the reward score of the non-preferred response sample. Then, the difference between the two is calculated as the reward score difference, which is used to reflect the relative quality of the training sample pair.

[0064] It should be noted that if the reward score of the preferred response sample is significantly higher than that of the non-preferred response sample, it indicates that the training sample is easy to distinguish; if the difference between the two scores is small, it indicates that the training sample has subtle semantic similarities or preferences, and is considered a more difficult sample.

[0065] More specifically, it can be predicted using the current reward model. and Receive reward score difference

[0066]

[0067] Where x represents the prompt message entered by the user; This indicates that the reward model compares the input cue x with the preferred response sample. The reward score obtained by predicting the combination of ; This indicates that the reward model considers the input cue x and the non-preferred response sample. The reward score obtained by predicting the combination of ; This represents the difference in reward scores between the i-th preferred response sample and the j-th non-preferred response sample. The reward score difference reflects the relative superiority or inferiority of the sample pair. The larger the value, the more obvious the advantage of the preferred response sample over the non-preferred response sample; the smaller the value, the more subtle the difference, indicating that they are a difficult-to-distinguish sample pair.

[0068] 103. Construct a cost matrix based on the difference in reward scores and the semantic correlation between preferred and non-preferred response samples.

[0069] In this embodiment of the invention, the cost matrix is ​​used to characterize the degree of difference between each preferred response sample and each non-preferred response sample. Specifically, the semantic correlation between the input prompt and its corresponding preferred and non-preferred response samples can first be obtained based on the text representations of the input prompt and its corresponding preferred and non-preferred response samples using semantic similarity calculation methods (such as cosine similarity, vector dot product, or representation similarity based on a pre-trained language model).

[0070] Subsequently, the reward score difference and semantic relevance are fused according to a preset combination to obtain the cost value. The cost value is used to comprehensively reflect the characteristics of a pair of samples in terms of both model prediction difference and semantic similarity: when the reward score difference is small and the semantic relevance is high, the cost value is large, indicating that the sample pair is a difficult sample that is hard to distinguish; when the reward score difference is large and the semantic relevance is low, the cost value is small, indicating that the sample pair is easy to distinguish.

[0071] Furthermore, the cost values ​​of all sample pairs can be arranged with the preferred response sample as the row index and the non-preferred response sample as the column index, thus constructing a cost matrix. This cost matrix provides the basis for subsequent calculation of the loss margin.

[0072] 104. Calculate the loss margin based on the cost matrix.

[0073] In this embodiment of the invention, the loss margin is used to dynamically adjust the discrimination strength of sample pairs during training. Specifically, for each preferred response sample, its corresponding row in the cost matrix can be obtained, where each cost value in the row represents the degree of difference between the preferred response sample and multiple non-preferred response samples; then, the cost values ​​in the row are aggregated according to a preset weighting method to obtain the loss margin corresponding to the preferred response sample.

[0074] When the preset weights are equal, the multiple cost values ​​of the row can be summed or averaged directly to obtain the loss margin of the preferred response sample.

[0075] In other implementations, the cost value can be weighted and summed based on the matching weights of optimal transmission to obtain a more globally optimal loss margin.

[0076] By calculating the loss margin based on the cost matrix, the training method can adaptively adjust the separation requirements according to the difficulty of the sample pairs: for difficult sample pairs with small difference in reward scores and high semantic similarity, the loss margin is larger, thus providing a stronger optimization signal in subsequent training; for easily distinguishable sample pairs, the loss margin is smaller, thus avoiding over-optimization of simple samples.

[0077] 105. Calculate the pairwise preference loss value with margin based on the loss margin, and update the parameters of the reward model to be trained with the goal of minimizing the loss value with margin, so as to obtain the trained reward model.

[0078] In this embodiment of the invention, the obtained loss margin can be combined with the reward score difference between the preferred response sample and the non-preferred response sample to correct the score difference, thereby obtaining a score difference with a margin. Subsequently, the score difference with a margin is input into a monotonically increasing probability mapping function to calculate the preference probability of the sample pair, and the loss value in the form of negative log-likelihood is calculated based on the preference probability.

[0079] Furthermore, the loss values ​​of multiple sample pairs can be weighted or averaged to obtain the overall pairwise preference loss value with marginality. To minimize this loss value, parameter optimization algorithms (such as gradient descent algorithms) are used to iteratively update the parameters of the reward model to be trained until the loss value converges or the number of updates reaches a preset number. Finally, the trained reward model is obtained, providing a stable reward signal for subsequent optimization of large language models.

[0080] In this embodiment of the invention, preference training sample pairs and a reward model to be trained are obtained. The preference training sample pairs include preferred response samples and non-preferred response samples. The reward score difference between the preferred response samples and the non-preferred response samples is calculated based on the reward model to be trained. A cost matrix is ​​constructed based on the reward score difference and the semantic correlation between the preferred response samples and the non-preferred response samples. A loss margin is calculated based on the cost matrix. A pairwise preference loss value with a margin is calculated based on the loss margin, and the parameters of the reward model to be trained are updated with the optimization objective of minimizing the loss value with a margin, thereby obtaining a trained reward model. By constructing a cost matrix based on the reward score difference and semantic correlation during training, and calculating the loss margin therefrom, and then using the loss margin to correct the loss value, the reward model can adaptively adjust the discrimination requirements according to the sample difficulty. This results in a clearer discrimination effect in sample scenarios with similar semantics but subtle differences, improving the model's learning ability for difficult samples and its overall generalization performance, avoiding over-reliance on simple samples, and thus improving the generation quality of large language models in complex tasks.

[0081] It is understood that in the specific implementation of this application, data related to preference training sample pairs and the like are involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required. Furthermore, the collection, use and processing of related data, as well as the training, construction and use of reward models and large language models, must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0082] Optionally, in the step of constructing the cost matrix based on the reward score difference and the semantic correlation between the preferred response sample and the non-preferred response sample, the reward score difference and the semantic correlation can be fused according to a preset combination method to obtain a cost value. The cost value is used to characterize the difference between the preferred response sample and the non-preferred response sample; the cost matrix is ​​constructed based on the cost value.

[0083] In this embodiment of the invention, the importance of semantic relevance and reward score difference can be balanced by setting a fusion coefficient. For example, a hyperparameter can be set. (Its value ranges from 0 to 1, with a preferred value of 0.5), and is used to control the weights of semantic relevance and reward score difference in the fusion process. When γ is larger, semantic relevance accounts for a higher proportion of the cost value; when γ is smaller, reward score difference accounts for a higher proportion of the cost value.

[0084] Specifically, semantic relevance and reward score difference can be numerically processed separately. Semantic relevance can be obtained using cosine similarity or other vector similarity methods; the reward score difference is obtained by performing a non-linear mapping on the score difference (e.g., using the sigmoid function) to obtain an intervalized result. Then, the two are weighted and combined according to a preset fusion coefficient γ to obtain the cost value corresponding to a single sample pair. Finally, the cost values ​​of all sample pairs are arranged with the preferred response sample as the row and the non-preferred response sample as the column to form a cost matrix, providing a basis for subsequent loss margin calculations.

[0085] More specifically, the construction of the above cost matrix can be further illustrated by the following formula:

[0086]

[0087] in, This represents the cost value between the i-th preferred response sample and the j-th non-preferred response sample, used to comprehensively characterize the degree of difference between the two;

[0088] This represents the semantic similarity between the i-th preferred response sample and the j-th non-preferred response sample; the larger the value, the closer the two are semantically.

[0089] This represents the difference in reward scores between the i-th preferred response sample and the j-th non-preferred response sample, the value of which is obtained by the reward model from predicting both of them separately;

[0090] This represents the result after nonlinear mapping of the reward score difference, preferably the output of the Sigmoid function, whose value is restricted to between 0 and 1, and is used to normalize the reward score difference;

[0091] The fusion coefficient represents the balance between the influence of semantic similarity and the difference in reward scores. It ranges from 0 to 1. When the value of γ is large, the proportion of semantic similarity in the cost value is higher; when the value of γ is small, the proportion of the difference in reward scores in the cost value is higher.

[0092] The cost value obtained in the above manner takes into account both the semantic similarity between response samples and the difficulty of the reward model in distinguishing between them, thus providing a more reasonable basis for subsequent loss margin calculation.

[0093] Optionally, in the step of calculating the loss margin based on the cost matrix, for each preferred response sample, the cost value corresponding to the preferred response sample as the row and multiple non-preferred response samples as the columns in the cost matrix can be determined; the cost values ​​are aggregated according to preset weights to obtain the loss margin corresponding to the preferred response sample.

[0094] In this embodiment of the invention, the preset weights can be equal weights. In this case, the loss margin can be obtained by summing or averaging the cost values ​​of the row. The above method is a point-to-point margin estimation method, which can achieve fast margin estimation with low computational cost.

[0095] Furthermore, in a specific implementation, the following baseline method can be adopted: For the i-th preferred response sample, its cost value is summed with the cost values ​​of all non-preferred response samples in the batch to obtain the loss margin. This method is denoted as "PointMargin," and its characteristics include a simple calculation process, no need for additional distribution matching solutions, and suitability for use in scenarios with limited computing resources or high real-time requirements.

[0096] More specifically, the calculation of the above-mentioned loss margin can be further explained by the following formula:

[0097]

[0098] in, This represents the loss margin corresponding to the i-th preferred response sample;

[0099] This represents the cost value between the i-th preferred response sample and the j-th non-preferred response sample;

[0100] B represents the number of non-preferred response samples included in a training batch.

[0101] A “batch” refers to a subset of data extracted from the preference dataset during model training. This subset contains several preferred response samples and non-preferred response samples, and is used for calculation in a single parameter update process.

[0102] Using the above method, the loss margin can be obtained directly by accumulating point-to-point cost values ​​without introducing complex distribution matching. This method is called PointMargin, and it is simple to calculate, suitable for quickly estimating sample difficulty and for use as a baseline method during training.

[0103] Optionally, in the step of calculating the loss margin based on the cost matrix, the optimal transfer used to characterize the matching relationship between the preferred response sample and the non-preferred response sample can also be obtained based on the cost matrix; for each preferred response sample, the loss margin corresponding to the preferred response sample is obtained by weighted aggregation based on the optimal transfer and the cost value in the cost matrix.

[0104] In this embodiment of the invention, optimal transmission can be achieved by constructing a first distribution of preferred response samples and a second distribution of non-preferred response samples. Under the constraint that the row sum and column sum of the transmission matrix are equal to the first and second distributions respectively, an optimization problem is established using the inner product of the cost matrix and the transmission matrix as the cost term. Furthermore, an entropy regularization term can be introduced into the cost term to ensure the sparsity and stability of the transmission results. By solving this problem using an optimization algorithm, the optimal transmission matrix used to characterize the matching relationship can be obtained.

[0105] Subsequently, for the i-th preferred response sample, the transmission weight vector corresponding to that sample can be obtained from the optimal transmission matrix, and combined with the cost value corresponding to that row in the cost matrix, weighted aggregation is performed according to the weights to obtain the loss margin of that preferred response sample. Compared with point-to-point aggregation, this method can comprehensively consider the globally optimal matching relationship between the preferred sample and multiple non-preferred samples when calculating the loss margin, thus more accurately reflecting the sample difficulty and improving the robustness and generalization ability of training.

[0106] Optionally, in the step of obtaining the optimal transmission based on the cost matrix to characterize the matching relationship between preferred and non-preferred response samples, a first distribution of preferred response samples and a second distribution of non-preferred response samples can also be constructed; a feasible region based on the first and second distributions can be determined, whereby the row edges of the transmission matrix are equal to the first distribution and the column edges are equal to the second distribution; within the feasible region, an optimization problem is established with the inner product of the cost matrix and the transmission matrix as the cost term and including an entropy regularization term; the optimization problem is solved to obtain the optimal transmission.

[0107] In this embodiment of the invention, the first distribution and the second distribution can be discrete probability distributions, such as uniform distributions, which are used to represent the overall constraints of the preferred response sample set and the non-preferred response sample set in the matching relationship, respectively. An entropy regularization term is used to improve the stability of the optimization process and avoid excessively sparse transmission results; specifically, this can be achieved by introducing a negative entropy weighted sum of the elements of the transmission matrix into the objective function. The optimization problem can be solved using iterative optimization algorithms, such as Sinkhorn iteration or other gradient-based numerical optimization methods, thereby efficiently obtaining the optimal transmission matrix that satisfies the constraints. .

[0108] Specifically, the first distribution mentioned above can be further explained by the following formula:

[0109]

[0110] Where P represents the first distribution of preferred response samples, which describes the distribution of preferred response samples within a training batch;

[0111] B represents the number of preferred response samples included in this training batch;

[0112] This represents a discrete distribution unit whose position corresponds to the combination of the i-th input cue and its preferred response sample. The value at this position is 1, and the value at other positions is 0.

[0113] The second distribution mentioned above can be further explained by the following formula:

[0114]

[0115] Where Q represents the second distribution of non-preferred response samples, which describes the distribution of non-preferred response samples within a training batch;

[0116] B represents the number of non-preferred response samples included in this training batch;

[0117] This represents a discrete distribution unit whose position corresponds to the combination of the j-th input cue and its non-preferred response sample. The value at this position is 1, and the value at other positions is 0.

[0118] The above optimization problem can be further explained by the following formula:

[0119]

[0120] in, This represents the optimal transmission result between the preferred response sample distribution P and the non-preferred response sample distribution Q;

[0121] T represents the transfer matrix, which describes the matching relationship between preferred response samples and non-preferred response samples;

[0122] C represents the cost matrix, whose elements are used to characterize the degree of difference between preferred and non-preferred response samples;

[0123] This represents the inner product of the transmission matrix and the cost matrix, used to represent the total cost under the matching relationship;

[0124] This represents the adjustment hyperparameter of the entropy regularization term, used to control the importance of the entropy term in the optimization problem;

[0125] The entropy value of the transfer matrix T is used to characterize the sparsity and stability of the matching relationship.

[0126] It is understandable that the left side of the equation is the optimization objective, and the right side represents an optimization problem. Its core is to solve the transmission matrix T so that the combination of transmission cost and entropy regularization term is minimized under the condition of satisfying the distribution constraints.

[0127] Alternatively, the optimization problem can be understood as follows: by introducing an entropy regularization term to balance the costs determined by the cost matrix C, a transmission scheme that reflects the difference between preferred and non-preferred sample pairs and is computationally more stable is obtained. Finally, solving this optimization problem yields the optimal transmission matrix. .

[0128] After obtaining the optimal transmission matrix (i.e., optimal transmission) Then, based on the cost values ​​in the optimal transmission and cost matrix, a weighted aggregation can be performed to obtain the loss margin corresponding to the preferred response sample. The specific calculation process can be further explained by the following formula:

[0129]

[0130] in, This represents the loss margin corresponding to the i-th preferred response sample;

[0131] The elements in the optimal transfer matrix represent the matching weights between the i-th preferred response sample and the j-th non-preferred response sample.

[0132] The elements in the cost matrix represent the degree of difference between the i-th preferred response sample and the j-th non-preferred response sample.

[0133] B represents the number of non-preferred response samples included in a training batch.

[0134] By weighted summing the matching weights in the optimal transfer matrix with the differences in the cost matrix, the loss margin for each preferred response sample can be obtained. This method reflects the global matching relationship between sample pairs from the perspective of the overall distribution, and more accurately reflects the sample difficulty compared to point-to-point accumulation.

[0135] Optionally, in the step of calculating the pairwise preference loss value with a margin based on the loss margin, the reward score difference after correction by the loss margin can also be calculated for each pair of preferred response samples and non-preferred response samples; the corrected reward score difference is input into a monotonically increasing probability mapping function to obtain the preference probability of the sample pair; the loss value in the form of negative log-likelihood is calculated based on the preference probability, and the loss values ​​of multiple preference training sample pairs are aggregated to obtain the pairwise preference loss value with a margin.

[0136] In this embodiment of the invention, the monotonically increasing probability mapping function can be the sigmoid function or other functions that satisfy probability normalization properties; the aggregation method of the loss values ​​can be weighted summation or averaging, so as to reflect the training error of all sample pairs within the batch as a whole. By introducing a loss margin in the calculation process, greater optimization pressure can be applied to difficult sample pairs that are semantically similar but have subtle differences, thereby enhancing the discriminative ability of the reward model in complex scenarios and effectively improving its generalization performance.

[0137] Specifically, the aforementioned pairwise preference loss with margin can be further explained by the following formula:

[0138]

[0139] Where L represents the pairwise preference loss with margin;

[0140] B represents the number of sample pairs in the training batch;

[0141] This represents the score of the reward model for the i-th preferred response sample;

[0142] This represents the score of the reward model for the i-th non-preferred response sample;

[0143] This represents the loss margin corresponding to the i-th sample pair, used to correct the score difference;

[0144] Probability mapping functions that represent monotonically increasing values, such as the Sigmoid function, are used to convert corrected fractional differences into probability values.

[0145] Therefore, the loss value is obtained by averaging the negative log-likelihood of all sample pairs within the batch. This can guide the reward model to distinguish between preferred and non-preferred responses during training, and improve the sensitivity to difficult samples by introducing a loss margin.

[0146] Understandably, the marginal The introduction of this feature enhances the reward model's ability to distinguish between preferred and non-preferred responses given an input x. Minimizing the reward score difference... This exceeds the margin, thus forcing a more pronounced separation between preferred and non-preferred responses. When The model will receive a stronger gradient signal (e.g., gradient magnitude). This guides it to focus on more difficult samples with small but positive margins. Larger margins Naturally, higher difficulty is associated with reward models, prompting them to focus on finer-grained distinctions and improve learning in the most critical areas.

[0147] In one embodiment, such as Figure 2 As shown, another method for training the reward model is also provided, which includes the following steps:

[0148] First, input the preference data and related hyperparameters as the starting conditions for training;

[0149] Subsequently, batches of triples were randomly sampled from the preference data as training samples to ensure the diversity and representativeness of the data.

[0150] Next, in the feature calculation stage, the reward score difference and semantic similarity are calculated respectively to characterize the difference between the preferred response sample and the non-preferred response sample;

[0151] Based on this, a cost matrix is ​​constructed to simultaneously reflect reward differences and semantic similarity. Then, the preferred response distribution and the non-preferred response distribution are used as inputs, and the optimal transport optimization problem is established by combining the cost matrix. The optimal transport plan is obtained by optimizing the solution, thereby capturing the differences between the two distributions in a principled manner.

[0152] Furthermore, based on the combination of optimal transmission and cost matrix, adaptive loss margin is calculated, making the training process more sensitive to difficult samples;

[0153] Finally, in the loss calculation and parameter update stage, the loss margin is introduced into the loss function to calculate the loss with margin, and the reward model parameters are continuously updated through gradient descent until the training process converges or reaches the preset number of iterations, thereby outputting the trained reward model.

[0154] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0155] In one embodiment, such as Figure 3 As shown, an optimization method for a large language model is provided, including the following steps:

[0156] 301. Obtain the trained reward model and the large language model to be optimized;

[0157] 302. Evaluate the output of the large language model to be optimized based on the trained reward model, and generate the corresponding reward signal;

[0158] 303. Use reward signals to guide policy optimization of the large language model to obtain an optimized large language model.

[0159] In this embodiment of the invention, the trained reward model is obtained by the reward model training method, and can accurately distinguish between preferred and non-preferred responses based on paired preference samples, thereby having a stronger generalization ability.

[0160] After receiving input prompts, the large language model generates candidate responses. The trained reward model scores or compares these candidate responses to obtain reward signals that characterize their quality, which are used to reflect the degree of fit between the output and the target output.

[0161] Specifically, policy optimization-based algorithms (such as Proximal Policy Optimization, PPO) can be used to iteratively update large language models, gradually making them more inclined to generate responses that better meet expectations, thereby improving the overall generation quality.

[0162] In one embodiment, a training apparatus for a reward model is provided, which corresponds one-to-one with the training method for the reward model in the above embodiments. For example... Figure 4 As shown, the training device for this reward model includes a first acquisition module 401, a first calculation module 402, a first construction module 403, a second calculation module 404, and a first update module 405. Detailed descriptions of each functional module are as follows:

[0163] The first acquisition module 401 is used to acquire preference training sample pairs and a reward model to be trained, wherein the preference training sample pairs include preferred response samples and non-preferred response samples.

[0164] The first calculation module 402 is used to calculate the difference in reward scores between the preferred response sample and the non-preferred response sample based on the reward model to be trained;

[0165] The first construction module 403 is used to construct a cost matrix based on the reward score difference and the semantic correlation between the preferred response sample and the non-preferred response sample;

[0166] The second calculation module 404 is used to calculate the loss margin based on the cost matrix;

[0167] The first update module 405 is used to calculate the pairwise preference loss value with margin based on the loss margin, and update the parameters of the reward model to be trained with the optimization objective of minimizing the loss value with margin, so as to obtain the trained reward model.

[0168] Optionally, the first building module 403 is further configured to:

[0169] The reward score difference and the semantic relevance are fused according to a preset combination method to obtain a cost value, which is used to characterize the difference between the preferred response sample and the non-preferred response sample;

[0170] Based on the cost values, the cost matrix is ​​constructed.

[0171] Optionally, the second calculation module 404 is further configured to:

[0172] For each preferred response sample, determine the cost value in the cost matrix corresponding to the preferred response sample as the row and multiple non-preferred response samples as the columns;

[0173] The cost values ​​are aggregated according to preset weights to obtain the loss margin corresponding to the preferred response sample.

[0174] Optionally, the second calculation module 404 is further configured to:

[0175] The optimal transmission, which characterizes the matching relationship between preferred and non-preferred response samples, is obtained based on the cost matrix.

[0176] For each preferred response sample, a weighted aggregation is performed based on the optimal transmission and the cost value in the cost matrix to obtain the loss margin corresponding to the preferred response sample.

[0177] Optionally, the second calculation module 404 is further configured to:

[0178] Construct a first distribution of the preferred response samples and a second distribution of the non-preferred response samples;

[0179] Determine a feasible region based on the first distribution and the second distribution, wherein the feasible region is used to constrain the row edges of the transfer matrix to be equal to the first distribution and the column edges to be equal to the second distribution;

[0180] Within the feasible region, an optimization problem is established with the inner product of the cost matrix and the transmission matrix as the cost term and including an entropy regularization term;

[0181] The optimal transmission is obtained by solving the optimization problem.

[0182] Optionally, the first update module 405 is further configured to:

[0183] For each pair of preferred response samples and non-preferred response samples, calculate the difference in reward scores after adjustment by the loss margin;

[0184] The corrected reward score difference is input into a monotonically increasing probability mapping function to obtain the preference probability of the sample pair;

[0185] The loss value in the form of negative log-likelihood is calculated based on the preference probability, and the loss values ​​of multiple preference training sample pairs are aggregated to obtain the pairwise preference loss value with margin.

[0186] Specific limitations regarding the training apparatus for the reward model can be found in the above section on the limitations of the training method for the reward model, and will not be repeated here. Each module in the aforementioned training apparatus for the reward model can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0187] In one embodiment, an optimization apparatus for a large language model is provided, which corresponds one-to-one with the optimization methods for the large language model in the above embodiments. For example... Figure 5 As shown, the optimization device for this large language model includes a second acquisition module 501, a first evaluation module 502, and a first optimization module 503. Detailed descriptions of each functional module are as follows:

[0188] The second acquisition module 501 is used to acquire the trained reward model and the large language model to be optimized, wherein the trained reward model is obtained by the training method of the reward model;

[0189] The first evaluation module 502 is used to evaluate the output of the large language model to be optimized based on the trained reward model and generate a corresponding reward signal.

[0190] The first optimization module 503 is used to use the reward signal to guide the strategy optimization of the large language model, so as to obtain an optimized large language model.

[0191] Specific limitations regarding the optimization apparatus for large language models can be found in the limitations of the optimization methods for large language models described above, and will not be repeated here. Each module in the aforementioned optimization apparatus for large language models can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0192] In one embodiment, a computer device is provided, which may be a terminal device, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, and network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes a readable storage medium storing computer-readable instructions. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer-readable instructions implement a training method for a reward model or an optimization method for a large language model. The readable storage medium provided in this embodiment includes both non-volatile and volatile readable storage media.

[0193] In this application embodiment, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor. When the processor executes the computer-readable instructions, it implements the steps of the training method of the reward model or the optimization method of the large language model as described above.

[0194] In one embodiment of the application, a readable storage medium is provided, which stores computer-readable instructions. When executed by a processor, the computer-readable instructions implement the steps of the training method of the reward model or the optimization method of the large language model described above.

[0195] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When executed, these computer-readable instructions can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0196] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0197] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for training a reward model, the method comprising: The method comprises: obtaining a pair of preference training samples and a reward model to be trained, the pair of preference training samples comprising a preferred response sample and a non-preferred response sample, the preferred response sample being a text response that is semantically complete, logically coherent, factually correct or task-compliant, and the non-preferred response sample being a text response that is semantically incomplete, logically incoherent, factually incorrect or task-incompliant; calculating a reward score difference between the preferred response sample and the non-preferred response sample based on the reward model to be trained; constructing a cost matrix based on the reward score difference and a semantic correlation degree between the preferred response sample and the non-preferred response sample; calculating a loss margin based on the cost matrix; calculating a pair-wise preference loss value with a margin based on the loss margin, and updating parameters of the reward model to be trained with a minimization of the pair-wise preference loss value with the margin as an optimization objective to obtain a trained reward model; the calculating of the loss margin based on the cost matrix comprises: solving an optimal transport for representing a matching relationship between the preferred response sample and the non-preferred response sample based on the cost matrix; for each preferred response sample, performing weighted aggregation based on the optimal transport and cost values in the cost matrix to obtain a loss margin corresponding to the preferred response sample; the solving of the optimal transport for representing the matching relationship between the preferred response sample and the non-preferred response sample based on the cost matrix comprises: constructing a first distribution of the preferred response sample and a second distribution of the non-preferred response sample; determining a feasible region based on the first distribution and the second distribution, the feasible region being used to constrain a row edge of a transport matrix to be equal to the first distribution and a column edge to be equal to the second distribution; establishing, in the feasible region, an optimization problem with an inner product of the cost matrix and the transport matrix as a cost item and containing an entropy regularization term; solving the optimization problem to obtain the optimal transport.

2. The method of claim 1, wherein the constructing of the cost matrix based on the reward score difference and the semantic correlation degree between the preferred response sample and the non-preferred response sample comprises: fusing the reward score difference and the semantic correlation degree in a preset combination manner to obtain a cost value, the cost value being used to represent a difference between the preferred response sample and the non-preferred response sample; constructing the cost matrix based on the cost value.

3. The method of claim 1, wherein the calculating of the loss margin based on the cost matrix comprises: for each preferred response sample, determining cost values corresponding to the preferred response sample as a row and a plurality of non-preferred response samples as columns in the cost matrix; aggregating the cost values in a preset weight to obtain a loss margin corresponding to the preferred response sample.

4. The method of claim 1, wherein the calculating of the pair-wise preference loss value with the margin based on the loss margin comprises: for each pair of preferred response sample and non-preferred response sample, calculating a reward score difference corrected via the loss margin; inputting the corrected reward score difference into a monotone increasing probability mapping function to obtain a preference probability of the sample pair; A loss value in a negative log-likelihood form is calculated based on the preference probability, and the loss values of multiple pairs of preference training samples are aggregated to obtain the pair-wise preference loss value with a margin.

5. An optimization method of a large language model, the method comprising: The method comprises: obtaining a trained reward model and a large language model to be optimized, the trained reward model being obtained by the reward model training method in any one of claims 1 to 4; evaluating the output result of the large language model to be optimized based on the trained reward model, and generating a corresponding reward signal; using the reward signal to guide the policy optimization of the large language model to obtain an optimized large language model.

6. A training device of a reward model characterized by, The device comprises: a first obtaining module configured to obtain a pair of preference training samples and a reward model to be trained, the pair of preference training samples comprising a preferred response sample and a non-preferred response sample, the preferred response sample being a text response that is semantically complete, logically coherent, factually correct, or meets task requirements, and the non-preferred response sample being a text response that is semantically incomplete, logically incoherent, factually incorrect, or does not meet task requirements; a first calculating module configured to calculate a reward score difference between the preferred response sample and the non-preferred response sample based on the reward model to be trained; a first constructing module configured to construct a cost matrix based on the reward score difference and a semantic correlation between the preferred response sample and the non-preferred response sample; a second calculating module configured to calculate a loss margin based on the cost matrix; a first updating module configured to calculate a pair-wise preference loss value with a margin based on the loss margin, and update parameters of the reward model to be trained to minimize the pair-wise preference loss value with a margin, thereby obtaining a trained reward model; The second calculating module is further configured to: obtain an optimal transmission representing a matching relationship between the preferred response sample and the non-preferred response sample based on the cost matrix; for each preferred response sample, perform weighted aggregation on the optimal transmission and cost values in the cost matrix to obtain a loss margin corresponding to the preferred response sample; The second calculating module is further configured to: construct a first distribution of the preferred response sample and a second distribution of the non-preferred response sample; determine a feasible region based on the first distribution and the second distribution, the feasible region being used to constrain a row edge of a transmission matrix to be equal to the first distribution and a column edge to be equal to the second distribution; establish an optimization problem with an inner product of the cost matrix and the transmission matrix as a cost term and containing an entropy regularization term within the feasible region; solve the optimization problem to obtain the optimal transmission. 7.A computer device, comprising a memory, a processor, and computer readable instructions stored on the memory and running on the processor, wherein, The processor executes the computer-readable instructions to implement the reward model training method in any one of claims 1 to 4 or the large language model optimization method in claim 5.

8. A readable storage medium, having stored thereon computer readable instructions, characterized in that, The computer-readable instructions are executed by the processor to implement the reward model training method in any one of claims 1 to 4 or the large language model optimization method in claim 5.

Images