Reinforcement learning method and device for large model

Abstract

The invention discloses a reinforcement learning method and device for a large model. In the method, a user question is input into a to-be-trained large model, multiple responses and confidence coefficients corresponding to the responses are obtained, and correctness rewards of the responses are determined according to correct answers of the user question. And for any response, performing weighting operation and cutting operation based on the confidence coefficient of the response and the correctness reward of the response to obtain the reward of the response. Wherein the weighting operation is used for using the reward of the high-confidence correct response greater than the reward of the low-confidence correct response, and the reward of the high-confidence error response is less than the reward of the low-confidence error response. The cutting operation is used for taking the lower bound value as the award when the award of the correct response is lower than the lower bound value and taking the upper bound value as the award when the award of the error response is higher than the upper bound value. Next, a model update based on a reinforcement learning algorithm is performed according to the plurality of responded rewards to update the large model. Privacy protection needs to be performed on training data used in the training process.

Description

Technical Field

[0001] This specification relates to the field of machine learning technology, and more particularly to a reinforcement learning method and apparatus for large models. Background Technology

[0002] Large Language Models (LLMs) have made groundbreaking progress in recent years, attracting widespread attention for their excellent performance on various complex tasks. However, to better align LLMs with human intent or specific task objectives, they typically enter the Reinforcement Learning (RL) phase after pre-training and instruction fine-tuning. In the RL phase, the quality of the reward signal is crucial to the model's learning effectiveness and stability. Privacy protection is also necessary when the sample data used during training contains private data.

[0003] Currently, there is a desire for improved solutions that can enhance training effectiveness and stability when performing reinforcement learning on large models, thereby improving the performance of large models. Summary of the Invention

[0004] This specification describes one or more embodiments of a reinforcement learning method and apparatus for large models, aiming to improve the training effect and stability when performing reinforcement learning on large models, thereby improving the performance of large models. The specific technical solution is as follows.

[0005] Firstly, the embodiment provides a reinforcement learning method for large models, including:

[0006] The user's question is input into the large model to be trained, and multiple responses and their corresponding confidence scores are obtained. The confidence scores are used to characterize the fluency and determinism of the corresponding responses.

[0007] The correctness reward is determined based on the correct answer to the user's question, and the correctness reward is used to indicate whether the corresponding response is a correct response or an incorrect response.

[0008] For any given response, a weighting operation and a pruning operation are performed based on the response's confidence level and its correctness reward to obtain the response's reward. The weighting operation uses the response's confidence level as the weight to weight the response's correctness reward, ensuring that the reward for a high-confidence correct response is greater than the reward for a low-confidence correct response, and the reward for a high-confidence incorrect response is less than the reward for a low-confidence incorrect response. The pruning operation uses the lower bound as the reward for a correct response when its reward is below a lower bound, and uses the upper bound as the reward for an incorrect response when its reward is above an upper bound.

[0009] Based on the rewards of the multiple responses, a model update based on a reinforcement learning algorithm is performed to update the large model.

[0010] In one implementation, the step of inputting the user question into the large model to be trained includes:

[0011] The user's question is input into the large model, and multiple responses are obtained;

[0012] The user question and the multiple responses are input into the version 0 large model to obtain the confidence scores corresponding to the multiple responses respectively; wherein, the version 0 large model is the large model before reinforcement learning is performed on the large model.

[0013] In one implementation, the step of inputting the user question and the plurality of responses into the version 0 large model includes:

[0014] The user question and the multiple responses are input into the version 0 large model, and the confidence of each text unit in any response is determined by the version 0 large model.

[0015] For any given response, calculate the geometric mean of the confidence scores of each text unit in that response, and use the geometric mean as the confidence score of that response.

[0016] In one implementation, the step of performing weighted and pruning operations based on the confidence level and correctness reward of the response includes:

[0017] The confidence level of the response is used as the weight to weight the correctness reward of the response, resulting in a confidence-weighted reward.

[0018] The confidence-weighted reward is pruned based on the lower and upper bounds to obtain the reward for the response.

[0019] In one implementation, the step of weighting the correctness reward of the response using the confidence level of the response as the weight includes: multiplying the confidence level of the response by the correctness reward of the response, and using the product as the confidence level-weighted reward.

[0020] In one implementation, the step of pruning the confidence-weighted reward based on the lower bound and the upper bound includes:

[0021] When the response is correct and the confidence-weighted reward is lower than the lower bound, the lower bound is used as the reward for the response.

[0022] When the response is an error and the confidence-weighted reward is higher than the upper bound, the upper bound is used as the reward for the response.

[0023] When the response is correct and the confidence-weighted reward is not lower than the lower bound, or when the response is incorrect and the confidence-weighted reward is not higher than the upper bound, the confidence-weighted reward is used as the reward for the response.

[0024] In one implementation, the correctness reward for a correct response is 1, the correctness reward for an incorrect response is -1, the confidence level ranges from 0 to 1, the lower bound is a value between 0 and 1, and the upper bound is a value between -1 and 0.

[0025] Secondly, the embodiments provide a reinforcement learning apparatus for large models, comprising:

[0026] The response determination module is configured to input a user question into a large model to be trained, and obtain multiple responses and their corresponding confidence scores. The confidence scores are used to characterize the fluency and determinism of the corresponding responses.

[0027] The correctness determination module is configured to determine the correctness reward corresponding to each of the multiple responses based on the correct answer to the user's question; the correctness reward is used to indicate whether the corresponding response is a correct response or an incorrect response;

[0028] The reward determination module is configured to perform a weighting operation and a pruning operation on any response, based on the confidence level and the correctness reward of the response, to obtain the reward for that response. The weighting operation uses the confidence level of the response as the weight to weight the correctness reward, such that the reward for a high-confidence correct response is greater than the reward for a low-confidence correct response, and the reward for a high-confidence incorrect response is less than the reward for a low-confidence incorrect response. The pruning operation uses the lower bound as the reward for a correct response when the reward is below the lower bound, and uses the upper bound as the reward for an incorrect response when the reward is above the upper bound.

[0029] The model update module is configured to perform a model update based on a reinforcement learning algorithm based on the rewards of the multiple responses, so as to update the large model.

[0030] Thirdly, the embodiments provide a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method described in any one of the first aspects.

[0031] Fourthly, an embodiment provides a computing device including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method described in any one of the first aspects.

[0032] In the methods and apparatus provided in the embodiments of this specification, the confidence level of the response is used to weight the reward for correctness. Among the obtained responses, the reward for a high-confidence correct response is greater than the reward for a low-confidence correct response, and the reward for a high-confidence incorrect response is less than the reward for a low-confidence incorrect response. This allows for the differentiation between completely correct and somewhat incorrect responses, as well as almost correct and completely incorrect responses, based on the reward, resulting in a richer reward signal. Simultaneously, through a pruning operation, the reward is limited to a reasonable range, preventing the model from learning low-confidence incorrect responses. This ensures that even when a large model is inaccurate in its own assessment of its capabilities, effective reward signals can still be used to guide the large model towards optimization in the correct direction. Therefore, the embodiments can improve the training effect and stability of reinforcement learning on large models, thereby improving the performance of large models. Attached Figure Description

[0033] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0034] Figure 1 This is a schematic diagram illustrating an implementation scenario of one embodiment disclosed in this application;

[0035] Figure 2 A flowchart illustrating a reinforcement learning method for large models is provided for an embodiment.

[0036] Figure 3 A schematic diagram illustrating the principle of determining a response reward as provided in this embodiment;

[0037] Figure 4 Experimental data graph showing the average response length of the large model output during the training process, provided for an example embodiment;

[0038] Figure 5 This is a schematic block diagram of a reinforcement learning device for large models provided for an embodiment. Detailed Implementation

[0039] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0040] Figure 1This is a schematic diagram illustrating an implementation scenario of one embodiment disclosed in this application. It includes a dataset, a computing device, and a large model. The dataset stores training data and contains multiple user questions and their corresponding correct answers. The computing device reads the user question and its corresponding correct answer from the dataset, inputs the user question into the large model, and determines multiple responses and their corresponding confidence levels for the user question using the large model. The computing device determines the correctness reward for each of the multiple responses based on the correct answer, and performs weighted and pruned operations on the correctness reward using the confidence level to obtain the reward corresponding to each of the multiple responses. The computing device uses this reward to update the parameters in the large model using a reinforcement learning algorithm.

[0041] Figure 1 This is merely one scenario for implementing the reinforcement learning method provided in this application. In practical applications, the method provided in this application can be applied to a variety of scenarios. For example, large models can be implemented on computing devices or on other devices besides computing devices, and datasets can be implemented on storage devices or computing devices.

[0042] The following explains the concepts and technical background involved in this application.

[0043] In this specification, "large model" refers to a large language model or large language model. A large language model is a natural language processing model based on deep learning techniques, typically with billions to hundreds of billions or even more parameters, possessing powerful language understanding and generation capabilities. Large language models can employ the Transformer architecture or its variants (such as GPT, BERT, etc.), which utilizes an attention mechanism to globally model sequential data, efficiently handling long-distance dependencies and thus performing exceptionally well in natural language tasks. Large language models learn the statistical features and semantic relationships of language through pre-training on large-scale corpora, giving them excellent generalization capabilities. The core capabilities of large language models include, but are not limited to: understanding contextual semantics, generating coherent and grammatically correct text, performing logical reasoning, and handling multi-task scenarios. Their usage typically includes two modes: direct inference and fine-tuning. In direct inference mode, the user guides the large language model to generate specific outputs by designing prompts. Prompts can be task descriptions or instructions in text form, used to stimulate the large language model's semantic understanding and generation capabilities. In fine-tuning mode, large language models are further trained on small-scale datasets within a specific domain to optimize their performance on specific tasks. The powerful generalization capabilities and flexibility of large language models make them an important tool in the field of artificial intelligence, providing efficient and accurate solutions for automated text generation and understanding, and enabling them to perform various natural language processing tasks such as text generation, question answering, and translation.

[0044] In some embodiments, large language models can also understand and generate data from other modalities (such as visual and audio). In this case, large language models can also be called multimodal large language models (MLLMs). MLLMs provide a richer and more natural interactive experience by integrating multiple types of input and output, such as text, images, and sound. The core advantage of MLLMs lies in their ability to process and understand information from different modalities and fuse this information to accomplish complex tasks. For example, MLLMs can analyze an image and generate descriptive text, or generate a corresponding image based on a text description. This cross-modal understanding and generation capability makes MLLMs widely applicable across multiple fields.

[0045] It should be noted that the key technologies of large language models can be found in the detailed description in the paper "A Survey of Large Language Models" (paper number: arXiv:2303.18223v16, published on March 11, 2025), and will not be repeated here.

[0046] For large models that have been pre-trained, or for large models that have been pre-trained and fine-tuned by instructions, reinforcement learning can be applied to further improve the performance of the large models.

[0047] Reinforcement learning (RL) is a machine learning paradigm. In it, an agent learns how to take better actions to maximize long-term cumulative rewards by interacting with the environment and receiving reward signals from it. RL is a machine learning method whose basic framework is the Markov Decision Process (MDP), which allows the agent to learn better policies through trial and error in its interactions with the environment. The agent performs actions in the environment and receives feedback based on the results of those actions, known as rewards or reward signals. These reward signals guide the agent to adjust its policy to maximize long-term cumulative rewards.

[0048] Reinforcement learning with verifiable rewards (RLVR) is a special form of reinforcement learning in which reward signals can be determined through automated verification mechanisms (such as code execution, logic checks, and knowledge base comparisons) without the need for manual annotation.

[0049] In this application, the large model is the agent to be trained, also known as the policy model (π). θ In reinforcement learning, π θ A large model represents the rules governing an agent's behavior; it is typically a deep neural network with parameters θ. It receives the environmental state as input and outputs various actions and their probability distributions. The response output by a large model is its output action. In the RLVR paradigm, reinforcement learning for large models faces numerous challenges.

[0050] It suffers from sparse and coarse-grained rewards. In the RLVR paradigm, binary, sparse rewards are typically provided (e.g., correct or incorrect answer), which makes it difficult for rewards to capture intermediate quality or subtle differences in the reasoning process of large LLM models. For example, a "almost correct" incorrect answer and a "completely incorrect" incorrect answer may receive the same low reward, making it difficult for large models to acquire fine-grained learning signals.

[0051] Furthermore, it also suffers from training stability issues. When the task difficulty is polarized—for example, too easy leading to all correct answers with no gradient, or too difficult leading to all incorrect answers with no gradient—sparse and coarse-grained reward signals are prone to vanishing gradients or instability. In deep learning, vanishing gradients refer to the phenomenon where gradient values ​​become very small during backpropagation, causing slow or even stagnant updates to model parameters.

[0052] It lacks oversight of the reasoning process. Its rewards focus solely on the final result, failing to guide large models in optimizing the quality of their internal thought processes or reasoning steps.

[0053] It also suffers from overthinking. Large LLM models sometimes exhibit overthinking when solving relatively simple problems, generating redundant intermediate inference steps, resulting in excessively long response sequences and consuming unnecessary computational resources and time. Overthinking refers to the phenomenon where large LLM models generate excessively long, redundant, or unnecessary inference steps when solving relatively simple problems, leading to reduced efficiency.

[0054] To address the aforementioned issues, this application provides a reinforcement learning method for large models. By introducing a novel reward mechanism, it aims to overcome the limitations of the RLVR paradigm, thereby improving the training efficiency, stability, and final performance of reinforcement learning for large model LLM.

[0055] The following is combined Figure 2 The embodiments are described in detail below.

[0056] Figure 2 This is a flowchart illustrating a reinforcement learning method for large models, provided as an example. The method is executed via a computing device, which can be any device, equipment, platform, or cluster of devices with computing and processing capabilities. The method includes the following steps. The method can be performed iteratively, and the following steps represent any single iteration.

[0057] Step S210: Input any user question x into the large model LLM to be trained to obtain G responses y and their corresponding confidence scores S.

[0058] Here, the user question x is the original question provided to the large model LLM by the user or environment at the start of training in this iteration. The large model LLM receives the input user question x and generates G responses y = {y1, ..., y2} based on the current model parameters or policy parameters θ. GG can be a positive integer, for example, G can be 8 or 16. Any response can be represented as text, containing multiple text units (tokens). The sequence length of the response is the number of text unit tokens it contains. A response can also be called an answer, that is, the answer to the user's question x.

[0059] In this embodiment, the large model LLM can be a large model that has been pre-trained and fine-tuned, or it can be a large model that has been pre-trained but not fine-tuned.

[0060] Each response y corresponds to a confidence level S. For example, the confidence level of response 1 is confidence level 1, and the confidence level of response 2 is confidence level 2. This confidence level can be represented as S(x, y), which characterizes the fluency and determinism of the corresponding response y. The confidence level S is a quantitative indicator of the degree of confidence of the large model LLM in the generated response y relative to the input user question x, and can also be called the conditional probability. The numerical range of the confidence level can be set to 0 to 1. In this embodiment, length normalization can be achieved by using the geometric mean of the conditional probabilities of the response sequence.

[0061] The G confidence levels of the G responses y can be determined by the large LLM model to be trained during the process of determining the G responses y, or they can be determined by the version 0 large model. The version 0 large model is the large LLM model before reinforcement learning; it is an observer, and its output has a natural, unbiased idea. However, after reinforcement learning, its conditional probabilities may be biased. Using the version 0 large model to determine the confidence levels of the responses can avoid biased estimations caused by the overestimation of the large LLM model after training, ensuring the objectivity of the confidence level assessment.

[0062] In practice, this step involves inputting the user question x into the large LLM model being trained, resulting in G responses y. Then, the user question x and the G responses y are input into the version 0 large model, which determines the confidence S corresponding to each of the G responses y.

[0063] In practical applications, a prompt word can be constructed from the user question x, G responses y, and task instructions. This prompt word is then input into a version 0 large model, which determines the confidence level of each text unit in any response. The prompt word comprises the user question x, G responses y, and task instructions. The task instructions instruct on determining the confidence level of the G responses y based on the user question x.

[0064] Regardless of whether it's a large LLM model under training or a large model based on version 0, it can be based on each response y i The confidence level of each text unit token contained in the response y is obtained. iThe confidence level is determined by the conditional probability of each text unit token, which is derived from the input user question x and the conditional probabilities of text unit tokens in the generated sequence. Determining the conditional probability of text unit tokens is an existing technique in large-scale model LLMs, and its details will not be elaborated further.

[0065] Specifically, for any response y i Calculate the response y i The geometric mean of the confidence scores of n text units (tokens) is used as the response y. i The confidence level. The geometric mean is the product of the conditional probabilities of n text unit tokens, raised to the power of n.

[0066] Geometric mean can more accurately reflect the overall response y of a large model. i The generation process ensures overall determinism, avoiding deviations from single tokens or lengths, and also avoiding the inherently low probability of long texts. Different responses have different sequence lengths, meaning the value of n will vary. The method described above, which uses the geometric mean to determine the confidence level, achieves the goal of length normalization for different response sequences, reflecting the overall confidence of the large model in the response sequence.

[0067] Step S220: Determine the correctness reward C for each of the G responses y based on the correct answer to the user's question x.

[0068] In this context, the correctness reward C represents whether the corresponding response is correct or incorrect. Generally, the correctness reward for a correct response is greater than that for an incorrect response. In reinforcement learning, this reward signal is used to encourage correct responses and penalize incorrect responses.

[0069] For example, the reward for a correct response is represented by a value of 1, and the reward for a wrong response is represented by a value of -1. This type of reward is a binary reward, denoted by the symbol C(y).

[0070] The binary correctness reward C(y) can be objectively judged by an external reward validator on the response y generated by the large model LLM, resulting in a judgment result of +1 (correct) or -1 (incorrect). In other words, the correctness reward C is an objective evaluation of whether a response is correct or not. That is, the user question x, G responses y, and the correct answer are input into the reward validator, which then obtains the correctness reward C for each of the G responses.

[0071] When the user's question x is a mathematical problem, its correct answer is a definite result. In this case, the reward validator can be a symbolic solver or a code executor that directly determines the correctness of the response y based on the correct answer.

[0072] When the user question x is a question in a question-answering task, such as open-ended questions or subjective tasks (e.g., essays, explanatory questions), the correct answer does not have mathematical certainty. In this case, the reward validator can be a knowledge base comparer that determines the correctness of the response y based on the semantic similarity between the correct answer and the response y. The output of the reward validator is a binary correctness reward C, where C(y) = ... i )=+1 indicates a correct response, while C(y) i A value of -1 indicates an incorrect response. The correctness reward C uses values ​​with this symmetry, such as 1 and -1, to represent correct and incorrect responses. This approach can more strongly penalize errors and also facilitates subsequent trimming operations.

[0073] Step S230, for any response y i Based on this response y i Confidence level S i And the response y i Correctness reward C i Perform weighted and pruned operations to obtain the response y. i Reward R i By processing the G responses in this way, a corresponding reward R can be obtained for each response.

[0074] The weighting operation is used to calculate the response y. i Confidence level S i As a weight, this response y i Correctness reward C i Weighting is applied. Higher confidence levels result in higher rewards than lower confidence levels, and the reward for a correct response is greater than the reward for an incorrect response. By weighting the correctness reward using confidence levels, the reward for a high-confidence correct response is greater than the reward for a low-confidence correct response, and the reward for a high-confidence incorrect response is less than the reward for a low-confidence incorrect response. This weighting method makes the correctness reward no longer simply represent correct or incorrect, but rather how correct it is and how many errors occurred, thus providing data on the degree of correctness. This enriches the reward signal.

[0075] The pruning operation is used when the reward for a correct response is below a lower bound e. correct The lower bound value e will be used. correct As the response y i Reward R i When the reward for an error response is higher than the upper bound e incorrect The upper bound value e will be set at that time. incorrect As the response y i Reward R i This pruning operation can prevent large models from updating in the direction of "low confidence error". Lower bound ecorrect and the upper bound value e incorrect This can be determined before training or before the current iteration. Generally, the lower bound value e... correct Greater than the upper bound e incorrect .

[0076] The following example illustrates a special case. When the task is difficult, i.e., when the user's question is hard to answer, most answers are wrong. In this case, even if some wrong answers have low confidence (e.g., S = 0.2), after standardization in reinforcement learning (subtracting the mean and dividing by the standard deviation, where the mean is a negative number with a large absolute value), they may become a positive advantage, causing the overall model to update in the direction of low-confidence errors.

[0077] The weighting and pruning operations in this step can be implemented in several ways. For example, the weighting operation can be performed first, followed by the pruning operation, or the confidence scores can be pruned first, followed by the weighting operation. The first method will be explained below.

[0078] In this implementation, a response y i Let's take the response y as an example. i Confidence level S i As a weight, this response y i Correctness reward C i The weighted average is then used to obtain the confidence-weighted reward R. i ′.

[0079] Specifically, the response y can be... i Confidence level S i With the response y i Correctness reward C i Multiply the products and use the resulting product as the confidence-weighted reward R. i For example, the confidence-weighted reward R can be obtained using the following formula. i ′:

[0080] R i ′=S i ·C i (1)

[0081] Alternatively, the product can be pre-processed, and the processed result can be used as the confidence-weighted reward R. i ′.

[0082] Next, based on the determined lower bound value e correct and the upper bound value e incorrect The confidence-weighted reward R i The response y is obtained by cropping the data. i Reward R i.

[0083] Specifically, when the response y i For a correct response, the confidence level can be weighted with a reward R. i ′ and lower bound value e correct Compare the values. Lower bound e correct Used to prune correct responses. The confidence-weighted reward R... i ′ below the lower bound e correct When, the lower bound value e is set. correct As the response y i Reward R i When the confidence level is used to weight the reward R i Not lower than the lower bound value e correct At that time, the confidence level is weighted by the reward R. i ′ as the response y i Reward R i "Lower than" can be understood as "less than" or "less than or equal to".

[0084] When the response y i When the response is incorrect, the confidence level can be weighted with a reward R. i ′ and the upper bound value e incorrect Compare. Upper bound value e incorrect Used for pruning error responses. When the confidence level is used to weight the reward R... i ′ higher than the upper bound value e incorrect When, the upper bound value e is set. incorrect As the response y i Reward R i When the confidence-weighted reward is not higher than the upper bound e. incorrect At that time, the confidence level-weighted reward R is calculated. i ′ as the response y i Reward R i .

[0085] Wherein, the lower bound value e correct It can be set to a value between 0 and 1, with an upper bound of e. incorrect It can be set to a value between -1 and 0. The lower bound is e. correct and the upper bound value e incorrect This can be called an adaptive pruning parameter, which is a hyperparameter. In practical applications, this pruning parameter can be dynamically adjusted based on the current training progress and model performance. Model performance can be, for example, validation set accuracy or average reward. Training progress refers to the number of iterations.

[0086] The above cropping operation can also be achieved using the Clipping operation, specifically according to the following formula:

[0087] (2)

[0088] Here, `e` is a hyperparameter, the value of which can be determined empirically, for example, it can be set to 0.2. `clip` is the function name. Through the above Clipping operation, the reward for a correct response can be limited to the range [1-e, 1], and the penalty for an incorrect response can be limited to the range [-1, e-1]. When `e` is 0.2, the reward for a correct response is limited to the range [0.8, 1], and the penalty for an incorrect response is limited to the range [-1, -0.8]. The lower bound is `e`. correct The value is 0.8, and the upper bound is e. incorrect The value is -0.8.

[0089] Figure 3 This is a schematic diagram illustrating an example of determining a response reward provided in an embodiment. The user question is "What's the weather like today?", and the four responses and confidence levels obtained from the large LLM model are as follows:

[0090] Temperature 25 degrees Celsius, sunny - confidence level 0.9;

[0091] The weather is nice today — confidence level 0.8;

[0092] Today is a good day for outings—confidence level 0.3;

[0093] Be happy no matter what — Confidence level 0.9

[0094] The correctness rewards for the four responses are +1, +1, -1, and -1, respectively. That is, the four responses are a correct response, a correct response, an incorrect response, and an incorrect response, respectively. After weighting the responses using confidence levels, the confidence-weighted rewards for the four responses are: +0.9, +0.8, -0.3, and -0.9, respectively. The following threshold e... correct The value is 0.8, and the upper bound is e. incorrect Taking a value of -0.8 as an example, by pruning the confidence-weighted reward, we can see that we only need to change the confidence-weighted reward from -0.3 to -0.8, and the pruned reward will be the final reward for each response.

[0095] The final rewards show that, after pruning, the correct response with high confidence (0.9) (the first response) receives a higher positive reward (+0.9), while the correct response with low confidence (0.8) (the second response) receives a lower positive reward (+0.8). The incorrect response with high confidence (0.9) (the fourth response) receives a larger negative reward (-0.9), while the incorrect response with low confidence (0.3) (the third response) receives a smaller negative reward (-0.8).

[0096] In one implementation, the confidence level S can be adjusted first to achieve the aforementioned pruning objective. Since the confidence level S ranges from 0 to 1, when the response y... i Confidence level S i When the confidence level S of the response is less than a preset threshold, i Modify to a preset threshold, based on the response y i Modified confidence level S i As a weight, this response y i Correctness reward C i We perform a weighted average to obtain the response y. i The modified confidence level is used only to determine the reward and not for any gradient backpropagation or model probability updates; that is, it is not used to directly update large models.

[0097] For example, the preset threshold mentioned above can take values ​​such as 0.8. When the response y i Confidence level S i If the confidence level is less than 0.8, directly increase it to 0.8. Then, execute R. i =S i ·C i The operation. When the response y i When the response is correct, C i =1, R i =0.8. When the response y i When it is an error response, C i =-1, R i =-0.8. This method can also achieve the above-mentioned cropping purpose.

[0098] The above pruning operation prevents incorrect answers with extremely low confidence from gaining a positive advantage due to their relatively good performance, thereby stabilizing the training dynamics and avoiding the model from falling into a dead end of perpetually low-confidence output errors. It is understandable that values ​​below the lower bound e will be pruned. correct The reward is raised to the lower bound e. correct The purpose is also to prevent situations where erroneous responses with extremely low confidence gain a positive advantage.

[0099] This step S230 uses a dynamic confidence-weighted reward and an adaptive pruning strategy. The dynamic confidence-weighted reward is a reward signal that combines the confidence score S(x, y) with the binary correctness reward C(y). The adaptive pruning strategy is a reward adjustment mechanism that dynamically adjusts the upper and lower bounds e of the pruning process. correct and e incorrect The reward R is restricted to a specific range to stabilize the training process.

[0100] Step S240: Based on the rewards R of the G responses y, perform a model update based on a reinforcement learning algorithm to update the large model LLM.

[0101] Specifically, the advantage function A can be calculated for each response y based on the G rewards R within this batch. The advantage function, in reinforcement learning, measures the relative advantage of a particular action (or response) compared to the average action (or average response) in that state. The G responses to a user question x can be considered a batch. The advantage function A can be calculated using the following formula:

[0102] A i =(R i -baseline) / standard deviation (3)

[0103] Among them, R i It is the i-th response y i The baseline is the average reward calculated based on G rewards R, and the standard deviation is calculated based on G rewards R. A i It is a reward R i The advantage function.

[0104] In this embodiment, R i It is a continuous value, not a binary reward signal. The dominance function is also not binary, but a continuous value within a certain range.

[0105] During step S240, the parameters θ in the large model can be updated based on the policy optimizer and multiple advantage functions. The specific update process can be found in existing techniques and will not be detailed here. The policy optimizer can be an optimizer implemented based on the GRPO algorithm.

[0106] GRPO (Group Relative Policy Optimization) is a reinforcement learning algorithm that can be used for large model LLM. Its characteristic is that it performs relative policy optimization on multiple responses generated by the model in a batch.

[0107] Steps S210-S240 above constitute one iteration. In practical applications, multiple user problems can be used, and the above iterative process can be executed repeatedly to continuously update the large LLM model. The iteration ends when the training termination condition is met. The termination condition may be, for example, reaching the maximum number of iterations or the performance of the large model converging.

[0108] In another embodiment of this application, in step S230, any response y can be... i , will respond y i Confidence level S iAs a weight, this response y i Correctness reward C i Perform a weighted operation to obtain the response y. i Confidence-weighted reward R i ′, and take it directly as the response y i Reward R i Processing the G responses in this way yields corresponding rewards R. In this embodiment, confidence-weighted rewards R are not applied. i The pruning operation is described in this embodiment. This example is applicable when the training data does not contain particularly difficult tasks, and the large model LLM obtains responses with relatively high confidence and no extremely low confidence responses. In this scenario, there will be no situation where an extremely low confidence incorrect answer has a positive advantage over other answers, therefore pruning is unnecessary.

[0109] Regardless of whether the task is particularly difficult, simple, or medium-difficulty, a pruning operation can be performed to prevent the model from learning towards low-confidence incorrect answers.

[0110] The correctness reward C obtained in step S220 is a binary sparse reward signal, and there are many shortcomings in using this reward signal for reinforcement learning of large LLM models. In this embodiment, confidence is used as a weight to weight the binary reward signal, which can overcome these shortcomings.

[0111] This implementation incorporates the confidence level S of the response into the correctness reward C of external validation, making the reward no longer a simple binary signal, but a continuous signal within both the correct and incorrect intervals. This allows for a fine-grained reflection of the large model's assessment of the intrinsic quality of the inference process. A correct answer with high confidence receives a higher positive reward, while a correct answer with low confidence receives a lower positive reward; a wrong answer with high confidence receives a larger negative reward, while a wrong answer with low confidence receives a smaller negative reward. This is the effect produced when confidence is used as the weight. This refined reward signal provides richer gradient information to the large LLM model, enabling it to distinguish between responses and inference paths of varying quality.

[0112] Incorporating the confidence level S into the correctness reward C also helps avoid gradient vanishing. Even when all answers are correct or all answers are incorrect, the reward signal differs depending on the confidence level. For example, a high-confidence correct answer receives a different reward signal than a low-confidence correct answer. This provides a non-zero gradient signal, which helps the model continuously learn and improve.

[0113] Pruning rewards can stabilize training. Adaptive reward pruning strategies are designed to address the instability that confidence-weighted rewards may introduce in certain scenarios. By limiting the reward value to a dynamically adjusted, reasonable range, it prevents the model from learning "low-confidence incorrect answers" (i.e., the model avoids high penalties by generating fuzzy incorrect answers). The pruning mechanism ensures that even when the model misjudges its own capabilities (poor confidence calibration), the reward signal remains stable and effective, guiding the model towards the correct optimization direction. Lower bound e correct and upper bound value e incorrect The adaptability further enhances this stability, enabling reward adjustment to better adapt to different training phases and model learning states.

[0114] When a large model misjudges its own capabilities—for example, if the model believes it has a 90% confidence level in answering a user's question (average confidence level of approximately 0.9), but the actual average confidence level is only 0.6, indicating the existence of low-confidence responses—pruning can increase the reward for correct responses with low confidence and decrease the reward for incorrect responses with low confidence. This allows for the use of effective reward signals to guide the large model towards optimization in the right direction.

[0115] Confidence reflects the fluency and certainty of a large LLM model in generating the entire response sequence, indirectly reflecting the quality of the inference process. When the confidence score S(x,y) is combined with the correctness reward C(y), it encourages the large model not only to give the correct answer but also to give it with higher confidence or a more fluent reasoning process. The training goal is to maximize the product of S and C, as a larger product means larger values ​​for S and C, thus encouraging the large model to not only give the correct answer but also the answer with higher confidence. This mechanism implicitly supervises the inference process, prompting the model to learn clearer and more accurate reasoning chains.

[0116] The reward signal in this embodiment includes a signal to suppress redundant inference. During the training of a large model, the large LLM model is encouraged to quickly converge to a high-confidence correct answer when faced with a solvable problem. Higher confidence in the answer avoids generating redundant intermediate steps or overexploration, thereby reducing the consumption of text unit tokens during inference. This effectively provides the model with an incentive signal to confidently and quickly provide the correct answer, thus improving inference efficiency and reducing overthinking.

[0117] The applicant used the method provided in this embodiment (i.e., confidence weighting and pruning operations) to conduct comparative experiments on multiple large-scale LLM models based on multiple datasets and multi-domain task datasets. These datasets include MATH, GSM8K, AIME24, and MMLU-Pro. The experimental results are shown in Tables 1 and 2.

[0118] Table 1

[0119]

[0120] Table 2

[0121]

[0122] Table 1 shows the accuracy data of the large model output after training. The large model used is Qwen2.5-3B. Table 1 lists the test accuracy data of the base large model, the large model trained based on the GRPO algorithm, and the large model trained using the method of the example (ConfClip) on various datasets.

[0123] Table 2 also shows the accuracy data of the large model output after training. The large model used is Qwen2.5-7B. Table 2 lists the test accuracy data of the base large model, the large model trained based on the GRPO algorithm, the large model trained using the method of the example (ConfClip), the large model trained based on the GSPO algorithm, and the large model trained using the method combining the GSPO algorithm and the method of this example (GSPO+ConfClip) on various datasets.

[0124] Group Sequence Policy Optimization (GSPO) is also a reinforcement learning algorithm. This algorithm aims to address the model collapse problem that often occurs when large models use existing reinforcement learning methods in the post-training phase; that is, the sudden and irreversible performance degradation of large models during training.

[0125] As can be seen from the experimental data in Tables 1 and 2, the method provided in this embodiment significantly improves the performance of large models. Compared to the baseline GRPO, the performance of the large model Qwen2.5-3B trained using the MATH dataset improved from 63.07% to 64.34%; the performance of the large model Qwen2.5-3B trained using the GSM8K dataset improved from 78.39% to 79.23%. Similar gains were maintained on the larger model (Qwen2.5-37B).

[0126] The method presented in this embodiment improves the generalization ability of large models. It also shows a slight performance improvement on the cross-domain dataset MMLU-Pro, indicating that it is not overfitting.

[0127] Figure 4 The experimental data graph shows the average response length of the large model output during the training process, as provided in the example. Figure 4These are the curves obtained from data fitting using the method of this embodiment and the comparison scheme (baseline GRPO) for the large model Qwen2.5-3B during the above experiments. The horizontal axis represents the number of iterations, and the vertical axis represents the length of the response. It can be seen that as training increases, the output length in the comparison scheme remains relatively constant. However, in the scheme of this embodiment, as training increases, the output length decreases relatively while improving accuracy. This indicates that the large model becomes increasingly confident in its answer during training and does not need to overthink or output additional unnecessary text unit tokens.

[0128] In this embodiment, the reward signal is more interpretable and also has high compatibility and ease of integration. The reward signal not only reflects the final result but also incorporates the model's evaluation of its own reasoning, providing a richer perspective for studying and debugging the behavior of large-scale LLM models. The method provided in this embodiment can be used to create a plug-and-play module, which can then be easily integrated into mainstream large-scale LLM reinforcement learning algorithms (such as GRPO, PPO, etc.) without requiring large-scale modifications to the underlying framework.

[0129] It is understood that before or during the use of the technical solutions of the various embodiments in this disclosure, users will be informed of the type, scope of use, and usage scenarios of the personal information involved in an appropriate manner in accordance with relevant laws and regulations, and user authorization will be obtained.

[0130] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose, based on the prompt message, whether to provide personal information to the software or hardware such as electronic devices, applications, servers, or storage media performing the operations of this disclosed technical solution.

[0131] As an optional but not limited implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0132] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0133] The foregoing description describes specific embodiments of this specification; other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than those shown in the embodiments, and the desired result may still be achieved. Furthermore, the processes depicted in the drawings do not necessarily need to follow the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0134] Figure 5 This is a schematic block diagram of a reinforcement learning device for large models provided as an embodiment. This device embodiment is related to... Figure 2 The method embodiment shown corresponds to this. The device 500 is deployed in a computing device and includes:

[0135] The response determination module 510 is configured to input a user question into a large model to be trained, and obtain multiple responses and their corresponding confidence scores, wherein the confidence scores are used to characterize the fluency and determinism of the corresponding responses.

[0136] The correctness judgment module 520 is configured to determine the correctness reward corresponding to each of the multiple responses based on the correct answer to the user question; the correctness reward is used to characterize whether the corresponding response is a correct response or an incorrect response;

[0137] The reward determination module 530 is configured to perform a weighting operation and a pruning operation on any response based on the confidence level and the correctness reward of the response to obtain the reward for the response. The weighting operation uses the confidence level of the response as the weight to weight the correctness reward of the response, such that the reward for a high-confidence correct response is greater than the reward for a low-confidence correct response, and the reward for a high-confidence incorrect response is less than the reward for a low-confidence incorrect response. The pruning operation uses the lower bound as the reward for a correct response when the reward is below the lower bound, and uses the upper bound as the reward for an incorrect response when the reward is above the upper bound.

[0138] The model update module 540 is configured to perform a model update based on a reinforcement learning algorithm according to the rewards of the multiple responses, so as to update the large model.

[0139] In one implementation, the response determination module 510 includes a response acquisition submodule 11 and a confidence acquisition submodule 12. The response acquisition submodule 11 is configured to input a user question into the large model to obtain multiple responses. The confidence acquisition submodule 12 is configured to input the user question and the multiple responses into a version 0 large model to obtain the confidence scores corresponding to each of the multiple responses. The version 0 large model is the large model before reinforcement learning is performed on it.

[0140] In one implementation, the confidence level acquisition submodule 12 includes a determination unit 12-1 and an averaging unit 12-2. The determination unit 12-1 is configured to input the user question and the multiple responses into a version 0 large model, and determine the confidence level of each text unit in any response using the version 0 large model. The averaging unit 12-2 is configured to calculate the geometric mean of the confidence levels of each text unit in any response, and use the geometric mean as the confidence level of the response.

[0141] In one implementation, the reward determination module 530 includes a weighting submodule 31 and a pruning submodule 32. The weighting submodule 31 is configured to weight the correctness reward of the response using the confidence level of the response as the weight, obtaining a confidence-weighted reward. The pruning submodule 32 is configured to prune the confidence-weighted reward according to the lower bound and the upper bound, obtaining the reward for the response.

[0142] In one implementation, the weighted submodule 31 is specifically configured to multiply the confidence level of the response by the correctness reward of the response, and use the resulting product as the confidence-weighted reward.

[0143] In one implementation, the pruning submodule 32 is specifically configured as follows: when the response is a correct response and the confidence-weighted reward is lower than the lower bound, the lower bound is used as the reward for the response; when the response is an incorrect response and the confidence-weighted reward is higher than the upper bound, the upper bound is used as the reward for the response; when the response is a correct response and the confidence-weighted reward is not lower than the lower bound, or when the response is an incorrect response and the confidence-weighted reward is not higher than the upper bound, the confidence-weighted reward is used as the reward for the response.

[0144] The above-described apparatus embodiments correspond to the method embodiments, and detailed descriptions can be found in the description of the method embodiments section, which will not be repeated here. The apparatus embodiments are derived based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments; detailed descriptions can be found in the corresponding method embodiments.

[0145] This specification also provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform... Figures 1 to 4 Any one of the methods described.

[0146] This specification also provides a computing device, including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement... Figures 1 to 4 Any one of the methods described.

[0147] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for storage media and computing devices are basically similar to the method embodiments, so they are described more simply; relevant parts can be referred to the descriptions of the method embodiments.

[0148] Those skilled in the art will recognize that the functions described in the embodiments of the present invention in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0149] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, or improvements made based on the technical solutions of the present invention should be included within the scope of protection of the present invention.

Claims

1. A reinforcement learning method for large models, comprising: The user's question is input into the large model to be trained, and multiple responses and their corresponding confidence scores are obtained. The correctness reward is determined based on the correct answer to the user's question, corresponding to each of the multiple responses. The correctness reward is used to indicate whether the corresponding response is a correct response or an incorrect response; For any given response, a weighting operation and a pruning operation are performed based on the response's confidence level and its correctness reward to obtain the response's reward. The weighting operation uses the response's confidence level as the weight to weight the response's correctness reward, ensuring that the reward for a high-confidence correct response is greater than the reward for a low-confidence correct response, and the reward for a high-confidence incorrect response is less than the reward for a low-confidence incorrect response. The pruning operation uses the lower bound as the reward for a correct response when its reward is below a lower bound, and uses the upper bound as the reward for an incorrect response when its reward is above an upper bound. Based on the rewards of the multiple responses, a model update based on a reinforcement learning algorithm is performed to update the large model.

2. The method according to claim 1, wherein the step of inputting the user question into the large model to be trained includes: The user's question is input into the large model, and multiple responses are obtained; The user question and the multiple responses are input into the version 0 large model to obtain the confidence scores corresponding to the multiple responses respectively; wherein, the version 0 large model is the large model before reinforcement learning is performed on the large model.

3. The method according to claim 2, wherein the step of inputting the user question and the plurality of responses into the version 0 large model comprises: The user question and the multiple responses are input into the version 0 large model, and the confidence of each text unit in any response is determined by the version 0 large model. For any given response, calculate the geometric mean of the confidence scores of each text unit in that response, and use the geometric mean as the confidence score of that response.

4. The method according to claim 1, wherein the step of performing weighted operations and pruning operations based on the confidence level and correctness reward of the response comprises: The confidence level of the response is used as the weight to weight the correctness reward of the response, resulting in a confidence-weighted reward. The confidence-weighted reward is pruned based on the lower and upper bounds to obtain the reward for the response.

5. The method according to claim 4, wherein the step of weighting the correctness reward of the response using the confidence level of the response as a weight includes: The confidence level of the response is multiplied by the correctness reward of the response, and the product is used as the confidence-weighted reward.

6. The method according to claim 4, wherein the step of pruning the confidence-weighted reward based on the lower bound and the upper bound includes: When the response is correct and the confidence-weighted reward is lower than the lower bound, the lower bound is used as the reward for the response. When the response is an error and the confidence-weighted reward is higher than the upper bound, the upper bound is used as the reward for the response. When the response is correct and the confidence-weighted reward is not lower than the lower bound, or when the response is incorrect and the confidence-weighted reward is not higher than the upper bound, the confidence-weighted reward is used as the reward for the response.

7. The method according to claim 1, wherein, The correctness reward for a correct response is 1, the correctness reward for an incorrect response is -1, the confidence level ranges from 0 to 1, the lower bound is a value between 0 and 1, and the upper bound is a value between -1 and 0.

8. A reinforcement learning device for large models, comprising: The response determination module is configured to input user questions into a large model to be trained, and obtain multiple responses and their corresponding confidence scores. The correctness judgment module is configured to determine the correctness reward corresponding to each of the multiple responses based on the correct answer to the user's question; The correctness reward is used to indicate whether the corresponding response is a correct response or an incorrect response; The reward determination module is configured to perform a weighting operation and a pruning operation on any response, based on the confidence level and the correctness reward of the response, to obtain the reward for that response. The weighting operation uses the confidence level of the response as the weight to weight the correctness reward, such that the reward for a high-confidence correct response is greater than the reward for a low-confidence correct response, and the reward for a high-confidence incorrect response is less than the reward for a low-confidence incorrect response. The pruning operation uses the lower bound as the reward for a correct response when the reward is below the lower bound, and uses the upper bound as the reward for an incorrect response when the reward is above the upper bound. The model update module is configured to perform a model update based on a reinforcement learning algorithm based on the rewards of the multiple responses, so as to update the large model.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-7.

10. A computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-7.

Images