Method, device and product for managing rewards

Abstract

A method, an apparatus, and a computer program product for managing rewards are presented. In the method, a first reward associated with a cue word and a response to the cue word are obtained through a reward model. The first reward indicates the matching degree between the cue word and the response. An implicit representation of the cue and the response is determined at a layer of a plurality of layers in the reward model. A degree of uncertainty of the first reward is obtained based on the implicit representation and an implicit representation space associated with the layer. A second reward is generated based on the first reward and the degree of uncertainty. By using the embodiments of the present disclosure, the reward output through the reward model can be more accurate by using the degree of uncertainty.

Description

Technical Field

[0001] The present disclosure relates generally to computer technology and, more particularly, to methods, apparatus, and computer program products for administering rewards. Background Art

[0002] Machine learning models, such as language models (LMs), are able to perform a wide range of natural language processing (NLP) tasks, including but not limited to text generation, summarization, translation, question answering, and sentiment analysis. Reinforcement learning with human feedback (RLHF) has been shown to be effective for aligning LMs with human preferences. RLHF typically involves three main steps. The first step is supervised fine-tuning (SFT) of the pre-trained LM using high-quality data. The second step is reward modeling to capture the human preferences that the LM should follow. The third step is policy optimization based on reinforcement learning (RL), in which the policy initialized from the SFT model is further improved by the reward model as an agent of human feedback in the RL-based policy optimization. However, in the above process, the reward model constructed from a limited data set of human preferences is usually unable to accurately represent the underlying human preferences. At this point, it is expected to accurately represent the underlying human preferences. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for managing rewards is provided. In the method, a first reward associated with a prompt and a response to the prompt are obtained through a reward model. The first reward indicates a match between a prompt word and a response. An implicit representation of the prompt word and the response at a layer in a plurality of layers in the reward model is determined. A degree of uncertainty of the first reward is obtained based on the implicit representation and an implicit representation space associated with the layer. A second reward is generated based on the first reward and the degree of uncertainty.

[0004] In a second aspect of the present disclosure, an electronic device is provided, comprising: a computer processor coupled to a computer readable memory unit, the memory unit comprising instructions, which, when executed by the computer processor, implement the method according to the first aspect of the present disclosure.

[0005] In a third aspect of the present disclosure, a computer program product is provided, the computer program product comprising a computer-readable storage medium having program instructions implemented therewith, the program instructions being executable by an electronic device to enable the electronic device to perform the method according to the first aspect of the present disclosure.

[0006] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The above and other objects, features and advantages of the present disclosure will become more apparent by describing some embodiments of the present disclosure in more detail in the accompanying drawings, in which like reference numerals generally refer to like components in the embodiments of the present disclosure.

[0008] Figure 1 An example environment for updating a LM according to machine learning techniques is shown;

[0009] Figure 2 An example diagram for managing rewards according to an embodiment of the present disclosure is shown;

[0010] Figure 3 An example diagram of a reward model according to an embodiment of the present disclosure is shown;

[0011] Figure 4 An example diagram of an implicit representation space according to an embodiment of the present disclosure is shown;

[0012] Figure 5 An example diagram illustrating the purpose of updating a language model according to an embodiment of the present disclosure;

[0013] FIG. 6A to FIG. 6D shows experimental results of the proposed solution according to an embodiment of the present disclosure;

[0014] Figure 7 An example flow chart showing a method for managing rewards according to an embodiment of the present disclosure; and

[0015] Figure 8 A block diagram of a computing device is shown in which various embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0016] The principle of the present disclosure will now be described with reference to some embodiments. It should be understood that these embodiments are described only for illustrative purposes, and help those skilled in the art to understand and implement the present disclosure, without implying any limitation on the scope of the present disclosure. The disclosure described herein can be implemented in various ways except the manner described below.

[0017] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0018] References in this disclosure to "one embodiment," "an embodiment," "an example embodiment," etc. indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment necessarily includes the particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same implementation. Furthermore, when a particular feature, structure, or characteristic is described in conjunction with an example embodiment, it is considered to be within the knowledge of those skilled in the art to be able to affect such feature, structure, or characteristic in conjunction with other embodiments, whether or not explicitly described.

[0019] It should be understood that although the terms "first" and "second" etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the scope of the example embodiments. As used herein, the term "and / or" includes any and all combinations of one or more listed terms.

[0020] The terms used herein are used only for the purpose of describing specific embodiments and are not intended to limit the example embodiments. As used herein, the singular forms "a", "an", "said", and "the" are also intended to include the plural forms unless the context clearly indicates otherwise. It will be further understood that the terms "comprises", "comprising", and / or "having" when used herein specify the presence of stated features, elements, and / or components, etc., but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof.

[0021] The principle of the present disclosure will now be described with reference to some embodiments. It should be understood that these embodiments are described only for illustrative purposes, and help those skilled in the art to understand and implement the present disclosure, without implying any limitation to the scope of the present disclosure. The disclosure described herein can be implemented in a variety of ways except for the manner described below. In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meanings as those of ordinary skill in the art to which the present disclosure belongs.

[0022] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of relevant laws and regulations and relevant rules.

[0023] It is understandable that before using the technical solutions disclosed in various embodiments of the present disclosure, the type, scope of use and usage scenarios of the personal information involved in the present disclosure should be notified to the user in an appropriate manner in accordance with relevant laws and regulations, and the user's authorization should be obtained.

[0024] For example, in response to receiving an active request from a user, a prompt message is sent to the user to explicitly inform the user that the requested operation will require obtaining and using the user's personal information. Therefore, the user can independently choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0025] As an optional but non-limiting implementation, in response to receiving an active request from the user, the method of sending the prompt information to the user may include, for example, a pop-up window, and the prompt information may be presented in the pop-up window in the form of text. In addition, the pop-up window may also carry a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0026] It is understandable that the above-mentioned process of notifying and obtaining user authorization is merely exemplary and does not limit the implementation of the present disclosure. Other methods that meet relevant laws and regulations are also applicable to the implementation of the present disclosure.

[0027] As mentioned above, in the process of updating LMs (especially large language models), the reward model often fails to accurately represent the underlying human preferences. For the sake of illustration, we can use Figure 1 The example environment in exemplifies the problem in reward modeling. Figure 1 An example environment 100 for updating a LM based on machine learning techniques is shown. Figure 1 As shown, the environment 100 may include a language model 120 that may output a response 122 based on a prompt word 110. Here, the prompt word 110 may be text or instructions provided to the language model 120 that are intended to guide the output or behavior of the language model 120. As an example, the prompt word 110 may be a simple question, description, instruction, or scenario designed to elicit a response 122 from the language model 120.

[0028] The environment 100 also includes a reward model 130 that can generate a reward 132 based on the prompt word 110 and the corresponding response 122 output by the language model 120. The reward 132 can indicate whether the response 122 matches the prompt word 110. For example, if the response 122 matches the prompt word 110, the value of the reward 132 can be high (e.g., 1), otherwise the value of the reward 132 can be low (e.g., 0). The reward 132 can then be used to update the language model 120 so that the language model 120 can output an updated response that better matches the prompt word.

[0029] However, reward models 130 built from limited datasets of human preferences often fail to accurately represent the underlying human preferences. Approximation errors exacerbated by distribution shifts during policy updates lead to unreliable rewards during the RL phase. This leads directly to the phenomenon of “reward over-optimization,” where the LM (as an example of the language model 120) exploits erroneous high-reward states, artificially inflating estimated proxy rewards while the true reward decreases. This misalignment causes the LM to prioritize reward maximization over actual content quality and user alignment, as seen in phenomena like “length bias,” where longer responses are erroneously favored regardless of their relevance.

[0030] Current mitigation strategies for over-optimization focus on penalizing samples with high reward uncertainty during RL-based policy training. These methods utilize an ensemble of different reward models trained with different seeds during the pre-training or fine-tuning phase, and use the variance of the estimated rewards in the ensemble to quantify uncertainty. While in theory, these uncertainty estimates can identify unreliable rewards, the computational costs associated with training and maintaining multiple reward models in memory during policy optimization make them impractical in real-world settings. Worse, given that the "scaling law" generally advocates larger reward models, this computational constraint also prevents achieving maximum performance potential.

[0031] In some related works, RLHF is a key method for fine-tuning language models to align with human preferences. However, the standard RLHF pipeline optimizes the policy towards an estimated reward model as a proxy for human feedback, which is susceptible to the over-optimization problem. This weakness leads to potential misalignment with true user preferences and subsequent performance degradation.

[0032] Several recent works aim to directly learn a policy model without RL optimization, either by learning the policy model directly from preference data without involving reward modeling, or by continuously simulating responses that are ranked high by the reward model. However, due to the inherent limitations of supervised learning, these methods face challenges in generalization and are particularly susceptible to data outside of the preferences.

[0033] In addition to methods to address over-optimization, recent works have proposed adversarial optimization frameworks to iteratively update reward and policy models. However, they utilize a min-max objective, where an inner optimization learns a policy to maximize the reward, while an outer minimization refines the reward model based on the provided golden preference data. Their inner optimization still directly relies on the estimated reward and thus suffers from the over-optimization problem. In contrast, the framework proposed in this disclosure adopts a max-min objective, where an inner minimization with a confidence region conservatively searches for the reward and then maximizes the policy based on the reward.

[0034] Before turning to the solution proposed by embodiments of the present disclosure to remedy the reward over-optimization problem in reinforcement learning from human feedback (RLHF), the present disclosure begins by providing an overview of the most commonly studied RLHF workflow. This can establish the notational and conceptual foundation necessary to understand the proposed solution. RLHF includes three main steps: supervised fine-tuning, reward modeling, and RL optimization.

[0035] RLHF usually starts with supervised fine-tuning (SFT), which fine-tunes a pre-trained model through supervised learning on high-quality samples from downstream tasks such as summarization or dialogue generation. The resulting model is denoted as π SFT .

[0036] The second stage of RLHF involves learning a reward model to pass the annotated preference data Capturing human preferences, where and Indicates the selected and rejected responses to prompt x i Assume that preferences are governed by some unknown reward model r following the Bradley-Terry (BT) model * (x,y) generates:

[0037] Typically, based on the model (usually π SFT ) to parameterize the reward model where an additional linear layer is added to the final transformer layer, which projects the final embedding layer to obtain a scalar reward. More specifically, let e(x,y): represents the final layer embedding of the prompt and response pair (x, y), and φ: represents an additional projection layer. The reward model is defined as in Including all adjustable parameters in φ and e(x,y). Given annotated preference data D, the reward model is minimized by minimizing the negative log-likelihood under the BT model. is trained to choose the answer y c Assign and reject answers r Compared to higher rewards:

[0038] In formula (1), σ represents a sigmoid function.

[0039] Finally, using the learned reward model to provide feedback during the RL policy optimization phase. Intuitively, the goal is to learn to make the reward Maximize without deviating from π SFT Too Far Strategy π θ :

[0040] In the above formula, β control and reference strategy π SFT bias, thereby maintaining a balance between reward maximization and adherence to SFT strategy behavior.

[0041] A significant limitation of RLHF is the fact that the RL optimization process relies on the estimated reward Rather than ideal or gold rewards * Although widely adopted, it ignores the potential for inaccuracies during reward model training. and r * Empirical solutions have reported that RL optimization tends to “manipulate” rewards so that when the estimated reward (i.e., proxy reward) increases, the ideal or golden reward decreases. This phenomenon is called over-optimization.

[0042] To alleviate this problem, several recent solutions propose to utilize an ensemble of K reward models in addition to the KL penalty in the original RL objective. Given a cue word x and its response y, these methods use the variance of rewards from different reward models to measure the uncertainty in the estimated reward, i.e. The reward is then penalized by uncertainty before being fed into policy optimization:

[0043] In Equation (2), γ controls the degree of uncertainty-based penalty. Intuitively, samples with high uncertainty are penalized during policy training to reduce the risk of over-optimizing an imperfect reward signal.

[0044] However, using reward ensembles, especially for LMs, results in significant computational overhead. This is due to the need to maintain multiple reward models in memory throughout the policy optimization process, where each model is a separate LM with billions of parameters. Worse, it imposes a limit on the upper bound of performance that can be achieved with given computational resources, since larger reward models are particularly powerful according to scaling laws. Therefore, it is desirable to use lightweight uncertainty estimates in reward modeling to handle the over-optimization problem without the need for heavy ensemble models.

[0045] In view of the above, this disclosure proposes a solution for managing rewards based on recent advances in understanding the representation learning capabilities in LMs, particularly in (multiple) internal layers (such as the last layer) of neural networks. Many previous solutions have shown that the internal layers contain key information about how the network predictions deviate from the original training data. With these findings, this disclosure begins by examining lightweight uncertainty estimation methods in reward modeling, relying only on internal layer embeddings of the reward model. Subsequently, this disclosure proposes Adversarial Policy Optimization (ADVPO) to address over-optimization by solving a distributed robust optimization problem around confidence intervals for reward model predictions for policy improvement.

[0046] Compared to exploiting uncertainty in prior work to address over-optimization, this approach has been shown to be a less conservative strategy for exploiting uncertainty in policy optimization. In various experiments, this disclosure shows how ADVPO can effectively mitigate reward over-optimization without the computational burden of ensemble-based methods. In addition, this disclosure shows that ADVPO generally produces improved policies as evaluated by human-assisted evaluation.

[0047] For more details on the solution for managing rewards refer to Figure 2 . Figure 2 An example diagram 200 for managing rewards according to an embodiment of the present disclosure is shown. Figure 2 As shown, a first reward 132 associated with a prompt word 110 and a response 122 to the prompt word 110 are obtained by a reward model 130. Here, the first reward 132 indicates the degree of match between the prompt word 110 and the response 122. An implicit representation 230 (sometimes also referred to as an embedding) of the prompt word 110 and the response 122 at a layer in a plurality of layers 210 to 220 in the reward model 130 is determined. The degree of uncertainty 240 of the first reward 132 (caused by the reward model 130) is obtained based on the implicit representation 230 and the implicit representation space associated with the layer. Then, a second reward (also referred to as an updated reward) is generated based on the first reward 132 and the degree of uncertainty 240. In an example, the second reward can be calculated by subtracting the degree of uncertainty 240 from the first reward 132. With these embodiments of the present disclosure, by correcting the first reward 132 with the degree of uncertainty, the reward output by the reward model 130 can be more accurate.

[0048] In an embodiment of the present disclosure, a layer may be an inner layer among multiple layers in the reward model 130. The implicit representation at the inner layer (especially the last layer) may affect the final output of the reward model 130. For more details about the last layer in the reward model 130, refer to Figure 3 . Figure 3An example diagram 300 of a reward model 130 according to an embodiment of the present disclosure is shown. Figure 3 As shown, the layers 210, ..., 212 in the multiple layers 210 to 220 before the last layer 220 can generate an implicit representation 230 as the input of the last layer 220. At this time, the layers 210, ..., 212 can act as encoders (which encode the prompt words and responses into the implicit representation 230), and the last layer 220 can act as a decoder (which decodes the implicit representation 230 into rewards). There is a connection between the last layer embedding and the reward uncertainty, and the last layer embedding of the reward model 130 encapsulates key information about the uncertainty in the model prediction.

[0049] As discussed, reward modeling can be decomposed into two parts: learning a good representation e(x,y) for a cue word and response pair via a pre-trained LM; and projecting the learned representation using a mapping φ to produce a scalar reward. Training LMs on a wide range of text corpora, combined with their large number of parameters, enables these models to develop versatile representations that can even be used in zero / few-shot tasks, demonstrating the adaptability of these representations.

[0050] However, the second part, which involves learning the projection weights φ, strictly relies on the preference data provided during the training of the reward model. Therefore, the reliability of the rewards is closely related to the accuracy and adaptability of the projection weights.

[0051] The above theory has been widely supported in previous solutions to deep neural networks. For example, one solution showed that the robustness of neural network models against these spurious correlations can be greatly improved by freezing the network to its last layer and retraining the projection head with only a smaller dataset (where the spurious correlations are not present). In the context of language models, recent experiments on weak to strong generalization further strengthened the theory. Their findings revealed that although the last layer embeddings of the LM are fine-tuned with noisy labels from weak supervision, the model can still perform well on subsequent classification tasks if the projection weights are accurately derived from the true labels. This highlights the adaptability that can be obtained with simple projection weights and the rich information encapsulated in the embeddings.

[0052] Building on the concept of generalized representations with dedicated projection weights, the present disclosure shifts the focus to the capabilities of the final layer for uncertainty estimation. The projection weights are rigorously estimated based on the preference data encountered during reward model training. Therefore, when a cue word and response pair is evaluated during the RL phase, the pair may deviate from what was observed during reward training (indicating that it has experienced distribution shift), thus making the predicted rewards unreliable as they may be outside the support range of the training set. Using these embodiments, the degree of uncertainty can be efficiently and effectively captured, and the computational cost can be reduced by relying only on the final layer embedding of the reward model.

[0053] It should be noted that the last layer is only an example of the layer. The layer may also include one or more layers (eg, the last two layers) or a single layer (eg, the last layer).

[0054] In the following, the disclosure may show how the last layer embedding of a reward model based on preference data (as an example of a reference sample for training a reward model) can be used as a feature map of the underlying kernel. This kernel then allows determining whether a new cue-word response pair is similar to the cue-word response pairs seen during training. If not, the corresponding uncertainty should be higher and therefore penalized during policy optimization.

[0055] In an embodiment of the present application, an implicit representation space may be obtained based on a plurality of reference samples for obtaining a reward model, wherein the reference samples in the plurality of reference samples include reference prompt words, reference responses, and reference rewards. In the example, the reference reward may be a value given by a person. For more details about the implicit representation space associated with a layer, refer to Figure 4 . Figure 4 An example diagram 400 of an implicit representation space according to an embodiment of the present disclosure is shown. Figure 4 As shown, the data set 410 may include a plurality of reference samples 412 to 414. Here, the data set 410 may be used for training Figure 1 The reward model 130 in FIG. The plurality of reference samples 412 to 414 may be mapped to the plurality of implicit representations 422 to 424, respectively, and the plurality of implicit representations 422 to 424 may form an implicit representation space.

[0056] In an embodiment of the present disclosure, the degree of uncertainty may be determined based on the deviation between the implicit representation and the implicit representation space. Figure 4, after acquiring the implicit representation space 420, a deviation 430 of the implicit representation 230 from the implicit representation space 420 may be calculated, and a degree of uncertainty may be determined based on the deviation 430. In an example, the degree of uncertainty may be proportional to the deviation 430. With these embodiments, a more accurate degree of uncertainty may be determined based on the calculated deviation, and thus a more accurate reward may be generated based on the degree of uncertainty.

[0057] In an embodiment of the present disclosure, the implicit representation space may be represented by a distribution of multiple reference implicit representations of multiple reference samples, and the deviation of the implicit representation may be represented by the implicit representation and the inverse distribution.

[0058] Many solutions for quantifying uncertainty in neural network outputs, guided by similarities derived from the model's final layer embeddings, have demonstrated their effectiveness in theory and in practice. Embodiments of the present disclosure provide the following two approaches (i.e., uncertainty quantification in neural bandit and Bayesian uncertainty modeling) because of their computational efficiency and theoretical soundness.

[0059] Regarding uncertainty quantification in neural bandits, with probability 1-δ, the following inequality holds regarding the degree of uncertainty or the width of the confidence interval (also known as the bias) of the estimated reward:

[0060] In formula (3), b is a function of δ (usually the smaller δ is, the larger b is), and e(x,y) T represents the implicit representation (i.e., embedding) of the prompt word and the response. In addition, M D (also known as the distribution of multiple reference implicit representations over multiple reference samples) summarizes all the last layer embeddings observed in the training data for the reward model, i.e. For ease of explanation, this disclosure indicates Intuitively, if a new cue-word response pair (x, y) is similar to an example in the training data, then applying the inverse M D This will lead to a small uncertainty Otherwise, the uncertainty will be high.Through these embodiments, the implicit representation space can be accurately represented by the distribution of multiple reference implicit representations, so that the degree of uncertainty can be accurately determined.

[0061] In an embodiment of the present disclosure, the implicit representation space may be represented by the mean and covariance of multiple reference implicit representations of multiple reference samples, and the deviation of the implicit representation may be represented by the implicit representation and the mean and covariance.

[0062] Bayesian uncertainty modeling, such as Bayesian linear regression (BLR) or Gaussian processes (GP), also provides an elegant way to quantify the degree of uncertainty in a closed form. To simplify the explanation, this disclosure first considers BLR and then describes GP. In BLR, the projection weights φ are assumed to follow a Gaussian prior, expressed as Where m 0 is the prior mean and S 0 is the prior covariance matrix.

[0063] Then, the predicted posterior mean μ of the reward for a new data point (x, y) in BLR is post and variance It can be expressed in closed form as follows: μ post =e(x,y) T m D (5)

[0064] In the above formula, ∈ represents accidental noise, S D and m D Denote the closed-form posterior mean and variance of the projection weights φ calculated based on the reward training data D. This disclosure has presented BLR, and it should be noted that for GP, it can be replaced by k(e(x,y),e(x',y')) by replacing each e(x,y) T e(x',y') is calculated to derive the mean and covariance similarly, where k is the kernel function. It should be noted that both uncertainty estimation methods described above are used to determine distances in embedding spaces with different scaled covariances.

[0065] This disclosure empirically studies the effectiveness of the two lightweight uncertainty estimation methods introduced above. Since the goal of this disclosure is to exploit uncertainty in reward modeling to alleviate the over-optimization problem in RLHF, this disclosure specifically examines whether the two methods can identify the difference between the estimated proxy reward and the golden reward during the RL stage. This disclosure extensively adopts a synthetic setting, in which a significantly larger "gold standard" reward model is trained, which simulates human preferences and provides labels for training the proxy reward model.

[0066] The present disclosure uses the reward and policy models established from LM to run the standard RLHF pipeline on an open dataset. During the proximal policy optimization (PPO) training phase, the present disclosure records the generated samples every 10 steps. Subsequently, the present disclosure calculates their golden rewards, proxy rewards, and reward uncertainties associated with the proxy rewards. The present disclosure compares the following uncertainty methods: CI (uncertainty quantified by formula (3)), GP (Gaussian process uncertainty by formula (6)), and other reward determination methods. Therefore, the lightweight uncertainty estimation method proposed in the present disclosure (i.e., CI and GP) is effective in adjusting over-optimization. The lightweight uncertainty estimation method proposed in the present disclosure is superior to reward sets with comparable parameter sizes.

[0067] In an embodiment of the present disclosure, a second reward may be generated based on the difference between the first reward and the degree of uncertainty. In an example, the second reward may be generated by subtracting the degree of uncertainty from the first reward. With these embodiments, by removing the degree of uncertainty, the updated degree of uncertainty may better capture human preferences.

[0068] Given that the lightweight uncertainty estimation methods (i.e., CI and GP) mentioned above capture the reliability of estimated rewards and show the potential to alleviate the over-optimization problem, an effective way to use them to improve policy optimization is designed below. The CI in formula (3) is mainly used to quantify the uncertainty in reward modeling.

[0069] Reward uncertainty enables the construction of confidence intervals that contain the golden reward with high probability, as defined in Equation (3). This motivates the formulation of a distributed robust optimization problem around confidence intervals of reward model predictions for policy improvement. Moving from simple point estimates of rewards to interval estimates, this approach prevents policy optimization from being misled by high rewards with high uncertainty, which is a major cause of over-optimization.

[0070] In an embodiment of the present disclosure, the layers in the reward model 130 may be updated based on the uncertainty degree 240. In the case where the layer is the last layer in the reward model, only the last layer may be updated, and the other layers in the reward may be frozen. Since the lightweight uncertainty estimation focuses only on the last layer embedding, the reward uncertainty (also referred to as the uncertainty degree) means the uncertainty of the learned projection weights in the reward model. Specifically, let and Represents the optimal and estimated projection weights of the last layer to predict the reward. This disclosure has: Recall that M. DAll last layer embeddings observed during the training of the reward model are summarized and can therefore be pre-computed. With these implementations, the reward model can output more accurate rewards and the reward over-optimization problem can be alleviated.

[0071] In an embodiment of the present disclosure, the response 122 may be generated by the language model 120 based on the prompt word 110, and the language model 120 describes the association relationship between the prompt word 110 and the response 122. In an example, the prompt word 110 may be a piece of text or instruction provided to the language model 120 as a starting point or context for generating the response 122. When provided with the prompt word 110, the language model 120 analyzes the text and attempts to generate a relevant and appropriate response 122. With these embodiments, the language model 120 is guided by the prompt word 110 to generate the response 122, so that the quality and appropriateness of the generated response 122 can be guaranteed.

[0072] In an embodiment of the present disclosure, the language model 120 can be updated based on the second reward. As described above, the problem in the RL optimization phase in the RLHF workflow is to update the language model 120 using an over-optimized reward (i.e., a first reward including the degree of uncertainty 240). Embodiments of the present disclosure can use a second reward (i.e., a reward from which the degree of uncertainty 240 has been removed) to update the language model 120, so the language model 120 can be updated based on an accurate reward. With these embodiments, the problem of over-reward can be alleviated, and the language model 120 can be updated based on a more important reward.

[0073] In an embodiment of the present disclosure, the language model 120 can be updated by determining the pessimistic reward corresponding to the prompt word and the response based on the degree of uncertainty through the reward model 130; and the language model 120 is updated based on the objective function associated with the pessimistic reward. Within the confidence interval of the learned projection weights, ADVPO searches for the most pessimistic reward prediction for the current strategy, and then improves the strategy for this reward. Since the golden reward is guaranteed to be covered with a high-probability confidence interval, using the most pessimistic reward (i.e., the lowest possible reward) naturally avoids over-optimization. Formally, this constitutes a distributed robust optimization problem. For more details on the optimization problem, refer to Figure 5 . Figure 5 An example diagram 500 of a goal 510 for optimizing a language model 120 according to an embodiment of the present disclosure is shown. Figure 5 As shown, the goal 520 of optimizing the reward model 130 may be included in the goal 510. An example of the goal 510 including the goal 520 may be as follows:

[0074] In formula (7), rφ (·) represents the reward obtained when using the projection weight φ while keeping the representation encoder unchanged. Using these embodiments, reward over-optimization can be avoided by using the most pessimistic reward, and the language model 120 can obtain better updates.

[0075] Furthermore, to better align adversarial search with how the reward model is obtained in Equation (1), the present disclosure also incorporates reference responses in our objective function via pairwise preference comparisons. A reference response can be any acceptable answer, such as annotated good responses from users or responses generated by an SFT model. Intuitively, incorporating a reference also prevents ADVPO from being overly conservative, as it ensures that the model is optimized along a direction that at least produces a better policy than where the reference was collected.

[0076] The inner minimization of Eq. (7) has a closed-form solution, so the optimization problem in Eq. (7) has an equivalent but more manipulable form:

[0077] In formula (8), and

[0078] The various steps of the present disclosure have been described above, and the experimental environment for implementing these steps will be described below. In the following, the present disclosure empirically evaluates the effectiveness of ADVPO. The experiments focus on evaluating whether ADVPO can alleviate the over-optimization problem; and whether ADVPO produces improved strategies in practice. The experiments use multiple widely adopted datasets for RLHF to conduct empirical investigations. For more details about the experimental results, refer to Figure 6A-6D .like Figure 6A-6D As shown, the gold reward is represented by the solid line, while the dashed line corresponds to the proxy reward. Figure 6B and Fig.6D The x-axis has a square root scale. Fig. 6A and Figure 6C Figure 2 shows how the golden reward (solid line) and proxy reward (dashed line) progress for the policy optimization step on multiple datasets, respectively. Figure 6B and Fig.6D captures the dynamics of the square root KL divergence, i.e. First, it should be noted that PPO (i.e., the existing solution) exhibits an over-optimization problem across both datasets, characterized by a significant rise in the proxy reward (dashed line), while the golden reward (solid line) begins to drop after reaching certain steps for multiple datasets. However, by conservatively taking the estimated proxy reward, i.e., using ADVPO (i.e., the proposed solution), the present disclosure can mitigate over-optimization for high but unreliable rewards, thereby ensuring that it remains within a reliable region (small KL divergence) with high golden rewards.

[0079] Additionally, the algorithmic performance of ADVPO and some baselines are then evaluated by evaluating the quality of responses generated from the resulting policies. Therefore, ADVPO can consistently outperform all baselines and produce improved policies.

[0080] In summary, this disclosure proposes Adversarial Policy Optimization (ADVPO), a novel approach designed to handle reward over-optimization in RLHF. First, this disclosure shows the effectiveness of lightweight uncertainty quantization that relies only on the last layer embedding. Subsequently, this disclosure proposes ADVPO to solve the distributed robust optimization problem around the confidence interval of reward model predictions for policy improvement. Empirical experiments on multiple datasets show that ADVPO effectively alleviates the over-optimization problem without incurring the computational burden associated with ensemble models. This produces improved policies in practical scenarios.

[0081] The above paragraphs have described the details for managing rewards. According to an embodiment of the present disclosure, a method for managing rewards is provided. For more details about the method, please refer to Figure 7 ,in Figure 7 An example flow chart of a method 700 for managing rewards according to an embodiment of the present disclosure is shown. At box 710, a first reward associated with a prompt word and a response to the prompt word are obtained through a reward model. The first reward indicates a match between the prompt word and the response. At box 720, an implicit representation of the prompt word and the response at a layer in a plurality of layers in the reward model is determined. At box 730, a degree of uncertainty of the first reward is obtained based on the implicit representation and the implicit representation space associated with the layer. At box 740, a second reward is generated based on the first reward and the degree of uncertainty.

[0082] In an embodiment of the present application, the layer is the last layer of the multiple layers in the reward model.

[0083] In an embodiment of the present application, determining the degree of uncertainty includes: obtaining the implicit representation space based on multiple reference samples used to obtain the reward model, reference samples in the multiple reference samples include reference prompts, reference responses, and reference rewards; and determining the degree of uncertainty based on the deviation between the implicit representation and the implicit representation space.

[0084] In an embodiment of the present application, the implicit representation space is represented by a distribution of multiple reference implicit representations of the multiple reference samples, and the deviation of the implicit representation is represented by the implicit representation and the inverse of the distribution.

[0085] In an embodiment of the present application, the implicit representation space is represented by means and covariances of multiple reference implicit representations of the multiple reference samples, and the deviation of the implicit representation is represented by the implicit representation and the means and covariances.

[0086] In an embodiment of the present application, generating the second reward includes: determining the second reward based on a difference between the first reward and the degree of uncertainty.

[0087] In the embodiment of the present application, method 700 further includes: updating the layer in the reward model based on the degree of uncertainty.

[0088] In the implementation manner of the present application, the response is generated by a language model based on the prompt word, and the language model describes the association relationship between the prompt word and the response.

[0089] In the implementation manner of the present application, method 700 further includes: updating the language model based on the second reward.

[0090] In an embodiment of the present application, method 700 also includes: updating the language model by the following operations: determining a pessimistic reward corresponding to the prompt word and the response based on the degree of uncertainty through the reward model; and updating the language model based on an objective function associated with the pessimistic reward.

[0091] According to an embodiment of the present disclosure, a device for managing rewards is provided. The device includes: an acquisition module, configured to acquire a first reward associated with a prompt word and a response to the prompt word through a reward model, wherein the first reward indicates the degree of match between the prompt word and the response; a determination module, configured to determine an implicit representation of the prompt word and the response at a layer in a plurality of layers in the reward model; an acquisition module, configured to acquire the degree of uncertainty of the first reward based on the implicit representation and an implicit representation space associated with the layer; and a generation module, configured to generate a second reward based on the first reward and the degree of uncertainty. The device also includes other modules configured to implement other steps in the above method.

[0092] According to an embodiment of the present disclosure, an electronic device for implementing method 700 is provided. The electronic device includes: a computer processor coupled to a computer-readable memory unit, the memory unit including instructions, which implement a method for managing rewards when executed by the computer processor. The method includes: obtaining a first reward associated with a prompt word and a response to the prompt word through a reward model, the first reward indicating a degree of match between the prompt word and the response; determining an implicit representation of the prompt word and the response at a layer in a plurality of layers in the reward model; obtaining a degree of uncertainty of the first reward based on the implicit representation and an implicit representation space associated with the layer; and generating a second reward based on the first reward and the degree of uncertainty.

[0093] In an embodiment of the present application, the layer is the last layer of the multiple layers in the reward model.

[0094] In an embodiment of the present application, determining the degree of uncertainty includes: obtaining the implicit representation space based on multiple reference samples used to obtain the reward model, reference samples in the multiple reference samples include reference prompts, reference responses, and reference rewards; and determining the degree of uncertainty based on the deviation between the implicit representation and the implicit representation space.

[0095] In an embodiment of the present application, the implicit representation space is represented by a distribution of multiple reference implicit representations of the multiple reference samples, and the deviation of the implicit representation is represented by the implicit representation and the inverse of the distribution.

[0096] In an embodiment of the present application, the implicit representation space is represented by means and covariances of multiple reference implicit representations of the multiple reference samples, and the deviation of the implicit representation is represented by the implicit representation and the means and covariances.

[0097] In an embodiment of the present application, generating the second reward includes: determining the second reward based on a difference between the first reward and the degree of uncertainty.

[0098] In the embodiment of the present application, method 700 further includes: updating the layer in the reward model based on the degree of uncertainty.

[0099] In the implementation manner of the present application, the response is generated by a language model based on the prompt word, and the language model describes the association relationship between the prompt word and the response.

[0100] In the implementation manner of the present application, method 700 further includes: updating the language model based on the second reward.

[0101] In an embodiment of the present application, method 700 also includes: updating the language model by the following operations: determining a pessimistic reward corresponding to the prompt word and the response based on the degree of uncertainty through the reward model; and updating the language model based on an objective function associated with the pessimistic reward.

[0102] According to an embodiment of the present disclosure, a computer program product includes a computer-readable storage medium having program instructions implemented therein, and the program instructions are executable by an electronic device to enable the electronic device to perform method 700 .

[0103] Figure 8 8 is a block diagram of a computing device 800 in which various embodiments of the present disclosure may be implemented. It should be understood that Figure 8 The computing device 800 shown in the figure is for illustration purposes only and does not imply any limitation on the functionality and scope of the present disclosure in any way. The computing device 800 can be used to implement the above-mentioned method in the embodiment of the present application. Figure 8 As shown, computing device 800 may be a general computing device. Computing device 800 may include at least one or more processors or processing units 810, memory 820, storage unit 830, one or more communication units 840, one or more input devices 850, and one or more output devices 860.

[0104] Processing unit 810 may be a physical or virtual processor and may implement various processes based on programs stored in memory 820. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to increase the parallel processing capabilities of computing device 800. Processing unit 810 may also be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller.

[0105] The computing device 800 typically includes various computer storage media. Such media can be any media that can be accessed by the computing device 800, including but not limited to volatile and non-volatile media, or removable and non-removable media. The memory 820 can be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (such as a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM) or flash memory), or any combination thereof. The storage unit 830 can be any removable or non-removable medium, and can include a machine-readable medium, such as a memory, a flash drive, a disk, or other medium, which can be used to store information and / or data and can be accessed in the computing device 800.

[0106] The computing device 800 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 8 Not shown in the figure, but a disk drive for reading from and / or writing to a removable non-volatile disk and an optical drive for reading from and / or writing to a removable non-volatile optical disk may be provided. In this case, each drive may be connected to a bus (not shown) via one or more data medium interfaces.

[0107] The communication unit 840 communicates with another computing device via a communication medium. Additionally, the functionality of the components in the computing device 800 can be implemented by a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the computing device 800 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or other general-purpose network nodes.

[0108] Input device 850 may be one or more of various input devices, such as a mouse, keyboard, trackball, voice input device, etc. Output device 860 may be one or more of various output devices, such as a display, speaker, printer, etc. With the aid of communication unit 840, computing device 800 may also communicate with one or more external devices (not shown) (such as storage devices and display devices), one or more of which enable a user to interact with computing device 800, or if necessary, any device (such as a network card, modem, etc.) that enables computing device 800 to communicate with one or more other computing devices. Such communication may be performed via an input / output (I / O) interface (not shown).

[0109] In some embodiments, instead of being integrated in a single device, some or all components of the computing device 800 may also be arranged in a cloud computing architecture. In a cloud computing architecture, components may be provided remotely and work together to implement the functions described in the present disclosure. In some embodiments, cloud computing provides computing, software, data access, and storage services, which will not require the end user to know the physical location or configuration of the system or hardware that provides these services. In various embodiments, cloud computing provides services via a wide area network (such as the Internet) using a suitable protocol. For example, a cloud computing provider provides an application via a wide area network that can be accessed through a web browser or any other computing component. The software or components of the cloud computing architecture and the corresponding data may be stored on a server at a remote location. The computing resources in a cloud computing environment may be merged or distributed at a location in a remote data center. Cloud computing infrastructure may provide services through a shared data center, although they act as a single access point for users. Therefore, cloud computing architecture may be used to provide components and functions described herein from a service provider at a remote location. Alternatively, they may be provided from a conventional server or directly installed or otherwise installed on a client device.

[0110] The functions described herein may be performed at least in part by one or more hardware logic components. For example, but not limited to, illustrative types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), etc.

[0111] The program code for executing the method of the subject matter described herein can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device so that the program code enables the functions / operations specified in the flow chart and / or block diagram to be implemented when executed by the processor or controller. The program code can be executed in whole or in part on the machine, in part on the machine as a stand-alone software package, in part on a remote machine, or in whole on a remote machine or server.

[0112] In the context of the present disclosure, a machine-readable medium may be any tangible medium that may contain or store a program used by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0113] In addition, although operations are shown in a particular order, this should not be understood as requiring such operations to be performed in the particular order shown or in sequence, or performing all operations shown to achieve desired results. In some cases, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be interpreted as limitations on the scope of the subject matter described herein, but should be interpreted as descriptions of features that can be specific to a particular implementation. Certain features described in the context of a separate implementation may also be implemented in combination in a single implementation. On the contrary, the various features described in a single implementation may also be implemented in multiple implementations individually or in any suitable subcombination.

[0114] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter specified in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

[0115] From the foregoing, it should be understood that specific embodiments of the presently disclosed technology have been described herein for illustrative purposes, but that various modifications may be made without departing from the scope of the present disclosure. Accordingly, the technology of the present disclosure is not to be limited, except as set forth in the appended claims.

[0116] The embodiments of the subject matter and functional operations described in this disclosure can be implemented in various systems, digital electronic circuits or computer software, firmware or hardware (including the structures disclosed in this specification and their structural equivalents) or a combination of one or more of them. The embodiments of the subject matter described in this specification can be implemented as one or more computer program products, that is, one or more modules of computer program instructions encoded on a tangible and non-temporary computer-readable medium, for being executed by a data processing device or controlling the operation of the data processing device. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a material combination that affects a machine-readable propagation signal, or a combination of one or more of them. The term "data processing unit" or "data processing device" covers all devices, equipment and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. In addition to hardware, the device can also include code that creates an execution environment for the computer program in question, for example, code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0117] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to execute on one computer or multiple computers located at one site or distributed over multiple sites and interconnected by a communications network.

[0118] For example, processors suitable for executing computer programs include general and special purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include or be operably coupled to receive data from or transmit data to or both of one or more large-capacity storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data. However, a computer need not have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices. The processor and memory can be supplemented by a dedicated logic circuit or incorporated into a dedicated logic circuit system.

[0119] The specification, together with the drawings, are intended to be considered exemplary only, wherein exemplary means example.As used herein, the use of "or" is intended to include "and / or" unless the context clearly dictates otherwise.

[0120] Although the present disclosure contains many details, these details should not be interpreted as limitations on the scope of any disclosure or content that may be claimed, but rather as descriptions of features that may be specific to a particular disclosed embodiment. Certain features described in the present disclosure in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations individually or in any suitable sub-combination. In addition, although features may be described above as acting in certain combinations and even initially claimed as such, in some cases, one or more features from the claimed combination may be removed from the combination, and the claimed combination may involve a sub-combination or a variation of the sub-combination.

[0121] Similarly, although operations are shown in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in sequence, or that all of the operations shown be performed, in order to achieve the desired results. In addition, the separation of various system components in the embodiments described in this disclosure should not be understood as requiring such separation in all embodiments. Only some implementations and examples are described, and other implementations, enhancements, and variations can be made based on what is described and shown in this disclosure.

Claims

1. A method for managing rewards, comprising: Acquire a first reward associated with a prompt word and a response to the prompt word through a reward model, wherein the first reward indicates a matching degree between the prompt word and the response; determining implicit representations of the cue word and the response at a layer in a plurality of layers in the reward model; Obtaining a degree of uncertainty of the first reward based on the implicit representation and an implicit representation space associated with the layer; as well as Based on the first reward and the degree of uncertainty, a second reward is generated.

2. The method of claim 1, wherein the layer is a last layer of the plurality of layers in the reward model.

3. The method of claim 1 , wherein determining the degree of uncertainty comprises: Acquiring the implicit representation space based on a plurality of reference samples used to acquire the reward model, wherein the reference samples in the plurality of reference samples include reference prompts, reference responses, and reference rewards; as well as The degree of uncertainty is determined based on a deviation of the implicit representation from the implicit representation space.

4. The method according to claim 3, wherein the implicit representation space is represented by a distribution of a plurality of reference implicit representations of the plurality of reference samples, and the deviation of the implicit representation is represented by the implicit representation and the inverse of the distribution.

5. The method of claim 3, wherein the implicit representation space is represented by means and covariances of a plurality of reference implicit representations of the plurality of reference samples, and the deviations of the implicit representations are represented by the implicit representations and the means and covariances.

6. The method of claim 1 , wherein generating the second reward comprises: The second reward is determined based on a difference between the first reward and the degree of uncertainty.

7. The method according to claim 1, further comprising: The layer in the reward model is updated based on the degree of uncertainty.

8. The method according to claim 1, wherein the response is generated by a language model based on the prompt word, and the language model describes the association relationship between the prompt word and the response.

9. The method according to claim 8, further comprising: The language model is updated based on the second reward.

10. The method according to claim 8, further comprising: Update the language model by: Determining, by the reward model, a pessimistic reward corresponding to the prompt word and the response based on the degree of uncertainty; as well as The language model is updated based on an objective function associated with the pessimistic reward.

11. An electronic device comprising a computer processor coupled to a computer readable memory unit, the memory unit comprising instructions that, when executed by the computer processor, implement a method for administering a reward, the method comprising: Acquire a first reward associated with a prompt word and a response to the prompt word through a reward model, wherein the first reward indicates a matching degree between the prompt word and the response; determining implicit representations of the cue word and the response at a layer in a plurality of layers in the reward model; Obtaining a degree of uncertainty of the first reward based on the implicit representation and an implicit representation space associated with the layer; as well as Based on the first reward and the degree of uncertainty, a second reward is generated.

12. The apparatus of claim 11, wherein the layer is a last layer of the plurality of layers in the reward model.

13. The apparatus of claim 11 , wherein determining the degree of uncertainty comprises: Acquiring the implicit representation space based on a plurality of reference samples used to acquire the reward model, wherein the reference samples in the plurality of reference samples include reference prompts, reference responses, and reference rewards; as well as The degree of uncertainty is determined based on a deviation of the implicit representation from the implicit representation space.

14. The apparatus of claim 13, wherein the implicit representation space is represented by a distribution of a plurality of reference implicit representations of the plurality of reference samples, and the deviation of the implicit representation is represented by an inverse of the implicit representation and the distribution.

15. The apparatus of claim 13, wherein the implicit representation space is represented by a mean and a covariance of a plurality of reference implicit representations of the plurality of reference samples, and the deviation of the implicit representation is represented by the implicit representation and the mean and the covariance.

16. The apparatus of claim 11, wherein generating the second reward comprises: The second reward is determined based on a difference between the first reward and the degree of uncertainty.

17. The apparatus of claim 11, the method further comprising: The layer in the reward model is updated based on the degree of uncertainty.

18. The device according to claim 11, wherein the response is generated by a language model based on the prompt word, and the language model describes the association relationship between the prompt word and the response.

19. The apparatus of claim 18, the method further comprising: The language model is updated based on the second reward, or the method further includes: updating the language model by: determining a pessimistic reward corresponding to the prompt word and the response based on the degree of uncertainty through the reward model; and updating the language model based on an objective function associated with the pessimistic reward.

20. A non-transitory computer program product, the non-transitory computer program product comprising a computer-readable storage medium having program instructions implemented therewith, the program instructions being executable by an electronic device to cause the electronic device to perform a method for administering a reward, the method comprising: Acquire a first reward associated with a prompt word and a response to the prompt word through a reward model, wherein the first reward indicates a matching degree between the prompt word and the response; determining implicit representations of the cue word and the response at a layer in a plurality of layers in the reward model; Obtaining a degree of uncertainty of the first reward based on the implicit representation and an implicit representation space associated with the layer; as well as Based on the first reward and the degree of uncertainty, a second reward is generated.

Images